Debugging the ASA build 435 errant TAT problem, found likely cause

[Lek767 37]

Greetings!I've sent the (long) message below to HiFi support via email. For the benefit of forum readers who are experiencing the same problem, and to collect more data that can corroborate (or refute) my findings, I'm cross-posting it here as well. I'm hoping it might be of some use.---------------------------------------- As you probably know, the HiFi Sim support forum currently has multiple threads describing a problem people are having with Active Sky Advanced: seemingly random excursions of the outside or total air temperature readings in mid-flight, resulting in intermittent engine performance degradation and an errant fight profile. I've been using ASA for the past 5 months and have been experiencing the same problems regularly. The remarks by your support manager Jim Skorna seem to suggest that you haven't been able to pinpoint the cause of this problem yet. Taking that as a challenge, I've tried to get to the bottom of this issue by conducting the experiments detailed below. While I haven't gone as far as reverse engineering ASA in order to find the exact piece of code that is at fault here, I think I've succeeded in narrowing down the scope sufficiently for you (having access to the source code, obviously) to locate the offending bug and to eradicate this vexing issue once and for all. Root cause summary: The errant temperature excursions (and other unwanted effects discussed further on) are caused by ASA sending illegal cloud layer parameters via SimConnect to FSX. I've been able to identify two specific cases of this happening: 1. cloud ceilings below ground level (negative elevation), and 2. cloud ceiling at ground level (zero elevation). In both cases, the offending values were the result of the feature called "advanced interpolation" in ASA. The illegal cloud layer parameters are then incorporated into a complete weather definition and sent to FSX via SimConnect. FSX then tries to parse this string of data, but aborts the process once it encounters the invalid cloud layer data. As a result, the rest of the weather definition, which is comprised of further cloud layers, temperatures and atmospheric pressure values (among others), is never parsed and gets discarded. This leads FSX to assume default values for these parameters: temperature 15C, dew point 12C, QNH 1013 mb. And this is what's causing the wild spikes in outside air temperature. Instead of receiving the actual temperature (say, minus 50C at a FL360), the aircraft is suddenly presented with an outside temperature of plus 15C! Understandably, the effects on the engines of a properly simulated aircraft (i.e. LDS 763, PMDG 747-400, MD-11, etc.) are dire and readily apparent. Note that other effects, such as the disappearance of precipitation and cloud layers, or a sudden change in atmospheric pressure, can also be ascribed to the same core issue. Testing setup: I've used two separate systems that are networked together: a laptop and a PC. Both are running Windows XP SP3. ASA build 435 is installed on the laptop. FSX SP2 is installed on the PC, as well as the LDS 763 and PMDG MD-11 aircraft models. A networked SimConnect link between the laptop and PC is set up per the ASA manual. On the laptop I'm also running the Wireshark network analyser. This enables me to capture all SimConnect traffic between laptop and PC and examine its contents. As a result, I'm able to determine how various types of weather as shown by the ASA GUI are encoded and presented to FSX via SimConnect instructions. ASA configuration: The ASA configuration used during these tests can be found in the attached "MyOptions.cfg" configuration file. While I haven't performed an exhaustive examination of the effects of every setting, some casual testing seems to suggest that the cause of this problem is not affected, or negated, by any specific configuration. Measurements, problem case number one: I conducted numerous test flights whilst collecting data, mostly over the Continental United States and the Alps in Europe. Once airborne, I just waited until the "TAT problem" presented itself; given enough cloud layers in the sky and the presence of lots of nearby weather stations, this usually didn't take long. As it occurred, I recorded the "Report" and "Wx Config" screens in ASA, as well as several readings of the actual SimConnect data being sent to FSX. Then, when the temperatures returned to "normal", I collected more data sets for comparison. I'll present the data of one of the test flights, which was a typical sample. This flight went from KMCI to KSFO on 26DEC2009, at FL340. The data shown were collected around 1215Z in the KFLA area. Note that there were major winter storms raging in the US at that time; I figured that this would increase the likelihood of running into the problem. Here are six SimConnect weather definitions sent to FSX as the "high temperature" problem occurred. The weather stations shown in the ASA Report screen during this segment were K72C, KFLA, KFTG and KBKF (among others): GLOB 261216Z 35928KT&D457OG 35928KT&A914NG 35928KT&A1829NG 35928KT&A2743OG 35928KT&A3658NG 35928KT&A5486NG 35928KT&A7315NG 35928KT&A9144NG 35928KT&A10363NG 35928KT&A11887NG 35928KT&A13411NG 35928KT&A14935NG 97KM&B-2000&D30000 6CU-022&CU000FLLS000N 8ST-011&ST000FOLS000N 8CU079&CU000FOLS000N M56/M66 M56/M66&A914 M56/M66&A1829 M56/M66&A2743 M56/M66&A3658 M56/M66&A5486 M56/M66&A7315 M56/M63&A9144 M56/M68&A10363 M56/M66&A11887 M56/M67&A13411 M56/M67&A14935 A2983 GLOB 261217Z 35728KT&D457NG 35728KT&A914NG 35728KT&A1829NG 35728KT&A2743NG 35728KT&A3658NG 35728KT&A5486NG 35728KT&A7315NG 35728KT&A9144NG 35728KT&A10363NG 35728KT&A11887NG 35728KT&A13411NG 35728KT&A14935NG 97KM&B-2000&D30000 6CU-026&CU000FOLS000N 8ST-015&ST000FOLS000N 8CU089&CU000FOLS000N M56/M66 M56/M66&A914 M56/M66&A1829 M56/M66&A2743 M56/M66&A3658 M56/M66&A5486 M56/M66&A7315 M56/M63&A9144 M56/M68&A10363 M56/M66&A11887 M56/M67&A13411 M56/M67&A14935 A2983 GLOB 261218Z 00028KT&D457NG 00028KT&A914NG 00028KT&A1829NG 00028KT&A2743NG 00028KT&A3658NG 00028KT&A5486NG 00028KT&A7315NG 00028KT&A9144NG 00028KT&A10363NG 00028KT&A11887NG 00028KT&A13411NG 00028KT&A14935NG 97KM&B-2000&D30000 6CU-026&CU000FOLS000N 8ST-015&ST000FOLS000N 8CU089&CU000FOLS000N M56/M66 M56/M66&A914 M56/M66&A1829 M56/M66&A2743 M56/M66&A3658 M56/M66&A5486 M56/M66&A7315 M56/M63&A9144 M56/M68&A10363 M56/M66&A11887 M56/M67&A13411 M56/M67&A14935 A2983 GLOB 261221Z 02028KT&D457OG 02028KT&A914OG 02028KT&A1829OG 02028KT&A2743NG 02028KT&A3658NG 02028KT&A5486NG 02028KT&A7315NG 02028KT&A9144NG 02028KT&A10363NG 02028KT&A11887NG 02028KT&A13411NG 02028KT&A14935NG 97KM&B-2000&D30000 7CU-036&CU000FOLS000N 7ST-024&ST000FNLS000N 6ST088&ST000FNLS000N M55/M65 M55/M65&A914 M55/M65&A1829 M55/M65&A2743 M55/M65&A3658 M55/M65&A5486 M55/M65&A7315 M55/M62&A9144 M55/M67&A10363 M55/M65&A11887 M55/M67&A13411 M55/M68&A14935 A2981 GLOB 261225Z 04829KT&D457LG 04829KT&A914OG 04829KT&A1829OG 04829KT&A2743NG 04829KT&A3658NG 04829KT&A5486OG 04829KT&A7315OG 04829KT&A9144NG 04829KT&A10363NG 04829KT&A11887NG 04829KT&A13411NG 04829KT&A14935NG 97KM&B-2000&D30000 7ST-035&ST000FOVR000N 8ST-025&ST000FNVR000N 7ST094&ST000FNVR000N M55/M65 M55/M65&A914 M55/M65&A1829 M55/M65&A2743 M55/M65&A3658 M55/M65&A5486 M55/M65&A7315 M55/M62&A9144 M55/M67&A10363 M55/M65&A11887 M55/M67&A13411 M55/M68&A14935 A2980 GLOB 261229Z 14536KT&D457OG 14536KT&A914NG 14536KT&A1829OG 14536KT&A2743NG 14536KT&A3658NG 14536KT&A5486OG 14536KT&A7315NG 14536KT&A9144NG 14536KT&A10363NG 14536KT&A11887NG 14536KT&A13411NG 14536KT&A14935NG 97KM&B-2000&D30000 4CU-038&CU000FOVR000N 8CU-028&CU000FOVR000N 7CI121&CI000FNVR000N M49/M59 M49/M59&A914 M49/M59&A1829 M49/M59&A2743 M49/M59&A3658 M49/M59&A5486 M49/M59&A7315 M49/M61&A9144 M49/M66&A10363 M49/M66&A11887 M49/M67&A13411 M49/M68&A14935 A2982 Then, when switching to the next station, KAPA, the problem vanished. The SimConnect data looked as follows:GLOB 261231Z 15229KT&D457NG 15229KT&A914NG 15229KT&A1829NG 15229KT&A2743NG 15229KT&A3658NG 15229KT&A5486NG 15229KT&A7315NG 15229KT&A9144NG 15229KT&A10363NG 15229KT&A11887NG 15229KT&A13411NG 15229KT&A14935NG 97KM&B-2000&D30000 5ST018&ST000FOVR000N 8ST038&ST000FNVR000N 5ST184&ST000FNVR000N M53/M63 M53/M63&A914 M53/M63&A1829 M53/M63&A2743 M53/M63&A3658 M53/M63&A5486 M53/M63&A7315 M53/M61&A9144 M53/M67&A10363 M53/M67&A11887 M53/M67&A13411 M53/M68&A14935 A2982 By reduction and deduction, I've managed to isolate the offending values. These are marked in red above, and of the form: 6CU-022&CU000FLLS000N I'm not an expert in SimConnect syntax, but if I'm not mistaken the "6CU-022" value indicates "6/8 cumulus cover at minus 2200 feet AGL". It is the "minus" part that causes FSX to choke and chortle. The parameters following that code carry temperature information (such as "M52/M63") and the altimeter reading ("A2982"). These are ignored after the parsing error occurs. As you can see, the KAPA weather has returned to "above ground" cloud layers only, and correspondingly the temperatures return to normal as well. Here are the KFLA Report and Wx Config screens at the moment the problem occurred: Note the "BKN-011" code in the Report screen, and that this data was generated by advanced interpolation. Note the "BKN00-" code in the METAR in the Wx Config screen, corresponding to the cloud layer shown to be at minus 1123 feet. Note also that *every* time this problem occurred, advanced interpolation had been the source of the offending data. Trying to replicate the problem via manual entry is not possible: the GUI prevents the user from entering negative altitudes. It is this very sanity check that seems to be missing from the advanced interpolation calculations. Measurements, problem case number two: A second example was collected whilst on the ground at KBOS, on 27DEC2009 around 1445Z. Here, the offending attribute *seems* to be a cloud layer at zero altitude AGL, although conceivably the coverage value of "clear, 0/8" could also be the cause. I've been unable to determine which of the two is the culprit, but am certain that it's either the one or the other. SimConnect data (problem occurring): GLOB 271354Z 13015KT&D457LG 13015KT&A914LG 13015KT&A1829NG 13015KT&A2743NG 13015KT&A3658NG 13015KT&A5486NG 13015KT&A7315NG 13015KT&A9144NG 13015KT&A10363NG 13015KT&A11887NG 13015KT&A13411NG 13015KT&A14935NG 3219&B-2000&D30000 0CU000&CU000FMLR000N 8CU001&CU000FLLR000N 8ST018&ST000FOLR000L 08/08 08/M04&A914 08/M07&A1829 08/M06&A2743 08/M16&A3658 08/M31&A5486 08/M42&A7315 08/M58&A9144 08/M69&A10363 08/M75&A11887 08/M73&A13411 08/M75&A14935 A2987 GLOB 271354Z 13015KT&D457NG 13015KT&A914NG 13015KT&A1829OG 13015KT&A2743MG 13015KT&A3658NG 13015KT&A5486NG 13015KT&A7315MG 13015KT&A9144NG 13015KT&A10363NG 13015KT&A11887NG 13015KT&A13411NG 13015KT&A14935NG 3219&B-2000&D30000 0ST000&ST000FNVR000N 8CU001&CU000FMVR000N 8ST018&ST000FOVR000N 1CI295&CI000FNVN000N 08/08 08/M03&A914 08/M05&A1829 08/M06&A2743 08/M16&A3658 08/M31&A5486 08/M41&A7315 08/M59&A9144 08/M68&A10363 08/M74&A11887 08/M73&A13411 08/M75&A14935 A2987 SimConnect data (problem gone, after removing the "CLR000" code from the Wx Config METAR and re-applying): GLOB 011537Z 13015KT&D457NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 3219&B-2000&D30000 8CU001&CU000RNVR000N 8ST018&ST000FNVR000N 08/08 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 A2988 GLOB 011548Z 13015KT&D457NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 13015KT&A0NG 3219&B-2000&D30000 8CU001&CU000FNVR000N 8ST018&ST000FNVR000N 08/08 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 08/M02&A0 A2988 Corresponding screenshots: Note that in this case, the source is Vatsim injection, rather than advanced interpolation. Also, no incorrect METAR codes are apparent on the Report screen. The offending code, "CLR000" is only present on the Wx Config screen. Deleting that one restores the situation to normal. As you'll likely surmise, I'm less confident about the exact mechanics of this particular occurrence than I am about the "negative altitudes" one. The latter is rather clear-cut, whereas this one seems a bit more elusive and prone to multiple interpretations. There is definitely something there, there, though. :-) Conclusion and recommendations: The vexing issue of erratic temperature swings, as well as some issues with popping cloud layers and QNH jumps, is caused by improper data validation in Active Sky Advanced, abetted by an interpolation algorithm that apparently is able to generate these improper data in the first place. A somewhat obtuse data parser in FSX compounds the problem by dropping whole sections of the weather definition string, resulting in wildly incorrect default values to be used instead. I would assume that, by looking into the advanced interpolation routines, the issue of negative cloud layer altitudes can be found and fixed. I'd also recommend adding output validation routines that puts the SimConnect output through one final check before sending it to FSX. I hope this overly long essay proves to be of use, and that you'll be able to fix this problem as a result. Feel free to contact me if you have any questions about my rambling. ;-) Kind regards,Lek.

[HUSSAR 272]

Nice work Lek. I have been a user and follower of ASA and have enjoyed it immensely. Any improvement makes an already great addon even better, thanks for your hard work.

[wombat778 0]

Lek, this is truly amazing debugging work. I have been following the TAT issue very closely, as it is the only thing holding me back from buying ASA. If they can nail this issue, I will be the first in line with my money...

[Bob Scott 7,721]

BINGO! A hat tip to you Lek.Now let's see what HiFiSim does with it. This problem has been festering a long time.Bob ScottColonel, USAF (ret)ATP IMEL Gulfstream II-III-IV-VColorado Springs, CO

[falk781 0]

Something to consider.... I have found that it definitley doesn't happen all of the time, but when it does, I found that if I immediately disable Direct Wind Control Smoothing (DWC) and then click "Save and Apply" on the bottom left, within about 2 or 3 seconds, the temperature immediately corrects itself. This works every single time. On the other hand if I am flying, and I enable DWC, then click "save and apply" the temperature immediately goes to +41.Does this help any?

[Clipper Ocean Spray 189]

The errant temperature excursions (and other unwanted effects discussed further on) are caused by ASA sending illegal cloud layer parameters via SimConnect to FSX. I've been able to identify two specific cases of this happening: 1. cloud ceilings below ground level (negative elevation), and 2. cloud ceiling at ground level (zero elevation).

Great, substantive post. I guess a related question to ask is, are the negative elevation cloud ceilings responsible for the cloud popping issue I (and others) continue to experience? That is, do occasional negative elevation clouds cause clouds to quickly disappear because FSX can't render such clouds, only to quickly reappear when the sim at a later point in time receives correct parameters?Another question, if it is ASA sending these illegal cloud parameters to FSX, then does ASA also send illegal cloud parameters to FS9 as well?

[Petraeus 147]

Excellent investigation, really great work.Your detection of ASA trying to generate clouds at negative and zero heights did make me wonder whether there was a relationship with the problem of very low clouds at high altitude airports. However, there does not appear to be. In the former case, negative or zero height values are being sent to FSX; in the latter case, positive values are being sent, only these positive values should be adjusted to take into account the airport elevation.Nevertheless, if HiFi Simulation do decide to correct the TAT problem, then they'll quite possibly be working in the same area of code, where cloud values are sent to FSX, and it'll also be an opportunity for them to correct the very low cloud problem.

[darem 84]

I sometimes get temperatures around 340

[Lek767 37]

Great, substantive post. I guess a related question to ask is, are the negative elevation cloud ceilings responsible for the cloud popping issue I (and others) continue to experience? That is, do occasional negative elevation clouds cause clouds to quickly disappear because FSX can't render such clouds, only to quickly reappear when the sim at a later point in time receives correct parameters?

That could well be the case, I think. In my experience, when flying along any particular track, the weather definition being sent to the sim can oscillate rapidly between the values of two different source weather stations sometimes. If one of those has an invalid cloud definition code in it, it would lead to the disappearance of all cloud layers whilst that particular station gets selected. The reason for that is that the cloud layer parameters are sent in order of increasing altitude; the errant ones with a negative or zero altitude are sent up front. As any additional parameters after that invalid code get dropped, this will disable the cloud layers altogether.

Another question, if it is ASA sending these illegal cloud parameters to FSX, then does ASA also send illegal cloud parameters to FS9 as well?

Sorry, I don't have FS9, so I wouldn't know if that version is being affected by the same issue as well.

[creppel 3]

Thanks for your work. I have that problem too and I hope HifSim can fix that soon.

[Fritz Bayerlein 0]

What is the short-term solution that us users can do to avoid this problem pending a permanent fix, Lek?

[jordanal 70]

Nice job Lek. Gotta love Wireshark! One of the most useful tools in the industry. I've used it before at work troublshooting bandwidth issues as well.

[Lek767 37]

What is the short-term solution that us users can do to avoid this problem pending a permanent fix, Lek?

Well, I've come up with a workaround that should relieve the worst symptoms. However, it's rather cumbersome to use and in a few cases isn't all that effective. I guess that's why they call it a workaround instead of a solution... Anyway, your mileage may vary, as usual.Disclaimer: as stated in my previous post, I only have FSX, not FS9. Therefore, I have no idea whether all this is relevant to FS9, and if so, if this workaround is in any way effective on that platform.Workaround for ASA invalid SimConnect METAR codes and resulting wacky weather (FSX)First, ascertain that you are, in fact, experiencing the issue that's being discussed in this thread. To do so, in the FSX menu bar select World -> Weather... Look at the information under Current Conditions: if the temperature is 15

[sbclark 55]

Looks like no comments from HiFi Sims yet?It would be great if this analysis helps them pin point the issue so a fix can be rendered. I have experienced this issue as well when my PMDG about fell out of the sky at cruise altitude due to these large excursions.Bill Clark

[jordanal 70]

Looks like no comments from HiFi Sims yet?Bill Clark

What concerns me a bit more is the fact that I don't recall a single post by Damian in ages. I hope he and his family are well. :(