I'll have continued updates here over time:
http://www.tropicalatlantic.com/update/
Although the below became a rather full update. Below I have some things I have not yet discussed.

Basically, I am rewriting the whole recon system. Eventually the system will get distributed to other sites, and the code that runs the current system is so ugly looking. There is already one site that is integrating the old system into their site, though I am not sure if it will be public yet, and in the future I want to make it easier. The new system will have a better looking archive. The interface will be a little more user friendly.

All the decoders are being rewritten. I need it so the code doesn't look ugly for people who want to run the system on their site. Some code can be condensed and written better which allows me to make improvements, like new features, much easier. So far the dropsonde and HDOB (URNT15) are the closest to being complete other than the new mapping and the features below. I was actually redoing some of the chart in the dropsonde decoder, so really, there is a massive amount of work redoing all that not to mention everything that follows. And of course the live system is being rewritten along with the code that does the Google Earth mapping, although the Google Earth files will actually look the same in the end. If someone used only my Google Earth product they would probably wonder what in the world took a few years when nothing changed.

None of the new system will be online until draft 1 is complete which I outline below. The code of the new system along with how it works is extremely different from the new system. The structure of the archive is changing behind the scenes. The location of every single thing will change. Previously I stored decoded data in the archive as HTML files. The new system decodes everything on demand and does not store HTML files. Why have an extra hundred thousand files, or however many it may be, lying around? This allows me to update everything at once when I make a change to the decoders, like add new features. (No need to redo all those HTML files this way, though if I made a change to how Google Earth data is done, I would need to regenerate all those files.) Of course the decoders have to always be backwards compatible so they can still process the old stuff which should not be too much of a problem as stuff rarely changes in these messages.

There may be various other miscellaneous updates in addition to the below.


Draft 1 - Late 2013:

• Storm Slice in HDOB messages, for Individual Messages, perhaps Automatic slices chosen out of a mission and perhaps the ability to choose your own slice - Tonight I am getting into this complex part that I made up the name for. For over half a decade I have wanted something that shows HDOBs graphically. Recon usually does a pass from one quadrant of the storm into another. Imagine a well stacked storm. As you come from one quadrant to the other the winds should increase as you get to the eye then decrease in the eye and then go up rapidly again and then decrease as you get further from the eye in the other quadrant. How about seeing that graphically? Imagine seeing a double wind maximum indicating a second outer eye wall starting to form. I might have the ability to graph two things at once. Imagine on the left side of the Y axis you have estimated SFMR surface wind (10s avg.) for example and then on the right side of the graph on the Y-axis you have flight level wind (10s avg.). You could have two bars graphed vertically. If the storm is not well stacked, the winds at the different levels may not go up and down at the same time when looked at visually. Another example would be flight level wind (10s avg.) on one side of the graph and extrap surface pressure in the other side, so we would have different scales needed on each side of the graph. You would have color 1 for flight level win and color 2 for extrap surface pressure vertical bars. You would alternate each on the graph with again the X axis being time of the ob. If the bars did not go up and down at the same time, the storm might not be well stacked. But this comes with assumptions. The storm slice feature is not as easy at it seems. (of course it couldn't be, it has to make my life difficult) You must know that the flight is traveling in a straight line through the eye! What if it does a loop? What if it turns in its pattern which it will always do at some point. What if the aircraft static air pressure changes and you go to a new level that will have different winds. The storm slice would then be misleading. So, you have to know what you are looking at. Here comes the hard part. I am going to try to determine if a storm slice is likely applicable. How? Good question, lol. I have some ideas. I need to see if the plane travels in a straight line. (approximately, using bearings) I need to know if the aircraft keeps about the same static air pressure. If it does, then I might display the storm slice automatically. If not, I might have it available but hidden and tell people why the system thinks it may not be as applicable. (They will have to look at a map and determine visually if it is applicable enough.) And here would be the really cool thing, but oh so difficult. In real time, generate storm slice of not just a single observation, but of an entire pass through a storm. This would be very complex. Imagine having labels for a slice like "Inbound into NW quadrant and outbound into SE quadrant" or something like that. Maybe only "Inbound into NW quadrant and into eye" if it then exited in some other direction or the mission was over and the aircraft rose to high altitude as indicated by aircraft static air pressure for example. It would be hard to do in real time, but awesome. Other methods could include having a slice for multiple obs someone adds into the manual decoder. Or, in the archive have a way for someone to pick the start and end time of the slice they want and then my site would process exactly what they want from the raw data I have. You could look at a map and see where the plane entered a quadrant and setup for a near straight line all the way through the eye and out to the other quadrant before the plane turned. Get that end time and enter it into the manual storm slice options and then get that whole pass graphed with whatever you wanted graphed, choosing perhaps the width of the bars on the graph, the height, etc. Not sure yet. In my model system I do a timeline of model data at the bottom of the archive page. Here is an example of 2012. That is pure HTML and CSS. Imagine that bar vertically showing wind speed on the Y axis on the left side (and perhaps another variable on the right side) and the time of the ob in each line of the HDOB message on the X axis. You could perhaps choose other variables, like from the code I am trying to play with tonight: 'extrap_surface_pressure', 'flight_level_wind', 'air_temp', 'dew_point', 'peak_flight_level_wind', 'sfmr_sfc_wind', and 'sfmr_sfc_rain'. It could take awhile, lol.


• Individual observations mapped in Google Maps - I have always wanted data in Google Maps, and this would be the first step to later doing it in the live system. For now, it would involve taking a single message and making it available in Google Maps just as it would appear in Google Earth. For HDOBs this would be most significant. The colored wind barbs would be there and clickable with a popup window just like Google Earth. (For IE8 and earlier there would be dots, no wind barbs. It was not until IE9, which you can't run on XP, that the code I am using became available in it. Firefox and Chrome work fine with the colored wind barbs.) You could add and remove obs from the message. Add and remove the track line. A screenshot from over a year ago when I last worked on this part. That is Google Maps, not Earth. No plugin and you can add and remove things. For other message types, I'm not sure what the mapping will be like. It would be a single point. Perhaps an icon you would usually see in Google Earth, or an option to just see a dot, which would be clickable with the decoded data for the ob. Not sure yet.


• NASA GlobalHawk dropsonde data - These dropsondes now come across in the same manner as other dropsondes from the Air Force and NOAA but my site's current recon decoder cannot yet decode them. I have already rewritten the new dropsonde decoder to allow these, but because I need to finish the entire system first, it remains unavailable to the public. The code is too different from the old dropsonde decoder and it should undergo more testing first. These will be in the new live system.


• NOAA AXBT data - I have already written a manual decoder to decode this, but this would then become part of the live system. NOAA AXBT data is in a format that is subject to change as it is new. AIr Force AXBT data is not available in real time, though I did have someone who was going to try and get it to me very delayed at times. Once I get further along I'll try that again.


• Better front page - A map of current recon will be on the front page along with some mission information from the current missions out along with links to those data pages.


• Live recon by aircraft - Previously you could view live product pages for recon. In addition to that you will be able to view by aircraft. Lets say there are two planes out and you want to follow one over the other. The new system will allow you to pick the aircraft you want to follow. It might just be by product, or I might even have a page where you can view all of the latest product types on the same page for that mission. (HDOB, sonde, vortex, RECCO, AXBT)


• Atlantic and East/Central Pacific basins all in one script - It's hard to explain how the current system works. You can add data in three ways into the site. The first is the live system, the second is the way an admin adds data manually and the third is for the public to add an ob from the NHC archive. The current system has duplicate systems for the East/Central Pacific basins and even one for the West Pacific basin that has not seen recon since several years ago (which I don't have the ability to have the public to add obs into also). That meant 8 different pieces of code that shared similar code that was all separate. The new system puts the shared code into one script. The new system will work for all these basins.


• Intelligent mode - The new system will likely operate every minute all the time. The system will try to determine when it should check for data more often. If recon has been active recently, it will check for data every minute. If not, when the system runs every minute it will simply exit if it has not been 5 minutes perhaps since the last actual check. There will be various intervals for different products to try to be most efficient.


• Redundancy - Intelligent mode will help a lot, but there will still simply be times where the system fails to catch an ob before the raw file for a particular product has a new ob. Sometimes recon is delayed because of technical issues. Then you have HDOBs released every minute to catch up. The system sometimes misses catching those obs. (Or there are two planes from the same agency releasing data within a minute or two of each other and that is a real common way my site misses data.) There are other unknown reasons why my host's server doesn't always get the data even though it should. I hope to have a redundant system in place. This system would check the NHC's recon archive for new data, perhaps every half hour or hourly. It would determine what data has come across since I last checked for data and download all that data. My site would then process all that, or perhaps I would make it intelligent enough only to process what has yet to be processed into the system. (Unfortunately I would have to download the data because I can't tell what the data is unless I download it.) In theory, this system could actually be run on some sites as the only live system. Most hosts do not allow you to operate a cron task, a way to have the script operate at a certain time, more often than every 10 to more often 15 minutes. The live system, as it is now, absolutely must operate every 10 minutes, but really needs to run every 5 minutes not to miss too much data and for the last year I run it ever 2 to 3 minutes. (2 then 3 then 2 then 3 etc) When recon is out, I reduce it to every minute. But for sites that can't run a quick cron task like that, they could use this system and actually run the recon system which has been a major hurdle for most sites.

Draft 2 - Late 2013 to early 2014:

• Live recon data in Google Maps - This would be the biggest new feature. It is going to take a lot of thinking to determine how best to do this in a manner that was most efficient. Coding it is not a problem if bandwidth was of no concern at all, but I need to think about trying to reduce the amount of work the site does in order to make this available in real time.


• Administrative Interface - I manage some things manually by FTP. The new system needs to be easier for other sites, so there will need to be a web based administrative interface for a site owner running the script to manage things.

Possibly longer term...

• Other graphs - Highest wind speeds from each mission, lowest surface pressure. Have graphs like how I plan to do the storm slice to see how the storm intensity was over time.


• New Google Earth visualization (basically storm slice in 3d in Google Earth) - I have debated about whether to try something that would be kind of interesting, yet not something of great use. In addition to having the Google Earth file I currently do, perhaps on demand if someone actually wanted to see it, have a file that would contain vertical bars to represent wind speed in place of the wind barbs. Imagine if you opened Google Earth and looked at something like a storm slice, where the higher the wind speed is the higher the vertical bar would be in the location where the wind was found. The more I think about it, it would basically be like storm slice, expect in 3d in Google Earth. You could choose what variable you wanted to show, just one at a time (otherwise it would look like a real mess), and then generate the file on demand. You could click the bar for other details like you would the other Google Earth file. This would be for HDOBs. No other product data would be in the file. As it is, it would mean a lot of bars everywhere as you would have to select or unselect what you wanted to see. I would probably make people click a folder that would display about 20 HDOBs at a time, what is usually in one message. That would allow them to slowly add the data they want. All would be very cluttered. Then they could pick out the obs to make a cross section of the storm. It's an idea. Interesting visually, but not sure how useful it would really be if I actually can do storm slice.


• NASA GlobalHawk track data - The UAV is so high up so this data is not extremely important. The dropsonde data is, which as discussed previously will be in the new system. The track data is hard to get access to and while at times I know where to get it, it would involve writing an entire live system dedicated to getting this data. Also, I would have to download a massive file of several megabytes each time I wanted to update the track and there is so much track data. I would only update the track every half hour or perhaps hour at best.


• Significant observations to a Twitter account for recon obs - I would be concerned about information not marked as suspect getting out and people getting the wrong idea, so I'm still not sure. There would be lots of disclaimers if I ever did this.