IOCP Key Point 0 © Rafa Leon Guillermo Buenadicha OPS OEX SMOS LEOP & SODAP OPERATIONS © Rafa Leon

Guillermo BuenadichaOPS OEX

IOCP Key Point 0

© Rafa Leon

SMOS LEOP & SODAP OPERATIONS

© Rafa Leon


SUMMARY

LEOP OPERATIONSPRESODAP OPERATIONS

SODAP OPERATIONSFUTURE OPERATIONS

CONCLUSIONSFOS CONFIGURATION

SOURCE FOS INPUTS and EADS DASA REPORTS


MIRAS LEOP OPERATIONS

LEOP phase comprised from Launch up to Payload Switch On, i.e. from the 02/11/09 to 17/11/09Main events for the payload in the period were:

Partial SW on, Arm deployment Thermal control through Arm Segment Heater LinesNominal Pointing of Instrument + ASHL control

Phase manned with teams (electrical, system, thermal) both at CNES and at ESAC.Main source of data, exception made of the one of Partial SW ON, was the Proteus E_HKTM. Files received at ESAC after each S_Band visibility.


MIRAS LEOP: PARTIAL SW ON

NIR TEMP

Hinge Temperatures (both for the hot and the cold hinges) within the model predictions and thermal range go ahead for deployment

PLPC capable of data acquisition and display within few minutes

OBT

LO LOCK


MIRAS LEOP: ARM DEPLOYMENT

ARM ON Pulse NOM ON Pulse RED OFF Pulse NOM NOM RELEASED OFF RED RED EOT SW 3 SW 3 travel EOT SW 2 SW 2 EOT SW 1 = ARM DEPLOYED RELEASED to DEPLOYED

A 18:00:51.696 18:00:51.946 18:00:52.446 0.750 18:00:52.446 18:00:52.946 1.00 18:03:39.446 167.0 18:03:55.196 182.750 18:04:12.446 200.0

B 18:00:53.196 18:00:53.446 18:00:53.946 0.750 18:00:54.071 18:00:54.446 1.00 18:03:31.946 157.875 18:03:47.571 173.5 18:04:06.946 192.875

C 18:00:54.696 18:00:54.946 18:00:55.446 0.750 18:00:55.446 18:00:55.946 1.00 18:03:30.696 155.250 18:03:46.446 171.0 18:04:04.571 189.125


MIRAS LEOP: THERMAL ASSESSMENT

Partial SW ON

DeploymentCycling ASHL SW ON 10 deg

SW ON 22 deg

NOM Attitude

ARM TEMPERATURES


MIRAS LEOP: NORMAL POINTING + ASHL

C2 Thermostat problem. For that segment, the nominal ASHL line did not trigger but the redundant did instead

ASHL Current Lines drained above foreseen limits (originally 4.2 amps for 4 segments).

C2 analyzed and reported in ARTS (SMOS-21). Monitored further after manoeuvres and no change. Suspected explanation is different set points for Nom and Red thermostats than the ones in specs. Close monitoring in case of use of Backup-1 configuration SW ON. no operational impact

Current Lines limits changed to 4.4 amps, no further triggering.


MIRAS PRESODAP

This phase went from Instrument SW On the 17/11/09 up to formal start of SODAP on the 23/11/09SW On implemented through 3 different visibilities, final 22 degrees setting done on 18/11/09 in the morning.Activities in the period:

SW ONPayload monitoring FCP-SDP-100First Data Acquisition Auto downlink mode left enabledCommanding tests

ITL managementInstrument commanding testOBOP test


MIRAS PRESODAP: PLM MONITORING

SW ON fully nominal, some minor remarksThe PLM TM SSC between 136 and 205 were not received inside S-Band packets. Packets were received in X_Band PUS TM. To be understood as an effect of the MIL BUS initialization.Received the event “652: Time_Correlator_Unexpected_UTC” with parameter "UTC_Received". Not seen in the partial SW ON, possibly due to the PLM reaction after first GPS received.During GPS absence it has been noticed that there are two values inside TM (3,25) that are not consistent.

SPC10107 PPS_ERROR_FLAG shows "valid" (row 0) all the timeDPC10107 UTC Cur Source shows "GPS" (raw 0) all the time

The TM value First Scrub Finished DMASME38 has a behaviour not understood (looks like wrong SW initialization). It triggers, however, at the end of first scrubbing.“Stripe” effect of first scrubbing now fully understood and nominal.


MIRAS PRESODAP: OBT CLOCK

The OBT has a (foreseen) dependency on temperature.

The OBT frequency is faster than the quoted 65536 hz (still meeting specs). This created problems in the first week to the PLPC time correlation algorithm (too frequent time correlation resets). Stability over time looks however good.

PLPC implements currently a workaround that allows a maximum deviation up to 10 msec. A permanent fix to characterise the clock slope is being developed. Caveat is that re-ingestion of the same data causes duplicates in different packet times. Also, under non nominal conditions (i.e. re ingestion of CCU reset data) the time grows exponentially, forcing a cleanup.


MIRAS PRESODAP: COMMANDING TESTS

In this phase the commanding to the PLM was verified, in an incremental manner.Driven by the FCP procedures FCP-SDP-xxxInitial direct and time tagged ping connection TC’sEnabling of the ITL queue and Thermal Settings reporting.Complex ITL management (insertion, partial and complete deletion, clearing)Instrument commanding test FCP-SDP-500, forcing generation of reports and exercising different PUS services On Board, including command rejections.OBOP test, exercising all the calibration OBOPs and also the cyclic functions On Board.All tests successful. No unforeseen TC failures.


MIRAS PRESODAP: ACQUISITION

The PLM autodownlink function was left enabled on purpose after SW ON.Therefore, ESAC XBAS was in charge of data acquisition.Autodownlink disabled on 19/11/09 before the afternoon passes.The passes on 20/11/09 were commanded using SPGF, in order to ensure proper data acquisition of the OBOP test for PLM data calibration purposes.All acquisitions were successful.The instrument mode was changed to Full Pol Mode on 20/11/09 to prepare a saturated Mass Memory for SODAP tests.


MIRAS PRESODAP: CAVA & CAKE MEETING


MIRAS SODAP OPERATIONS

Already covered and defined by M. Zundo.FOS support to Mission Planning generation through SPGFFOS support to sequence of commanding, since most of the SODAP tests involved later manual modification of the SPGF planning outputsSupport to analyze the data downlink statisticsSupport to analysis of data gaps and identification of problemsReorientation of some tests to re-dump given mass memory locations for MM P3 analysis.No unforeseen command failures in the period.The mission planning and the command generation in the first 2 weeks of the period were based on a daily cycle. Extra S_Band passes provided by CNES to ensure it.


MIRAS SODAP OPERATIONS: PLM ANOMALIES

Even though some pertain to PRESODAP period, they were all characterised and analyzed during SODAP.

RAM single bit errors (3)MM single bit errors (8)CMN Unlock (6)CMN A2 drift (3)Mass Memory Partition 6 (7th) Latch Up (1)MM Partition 3 (4th) gap problems

RAM and MM single bits will be monitored along mission for radiation statistics, no clear correlation with geographical areas or weaker MM zones.


MIRAS SODAP OPERATIONS: SEU SUSPECT ANOMALIESMM Single Bit errors

RAM Single Bit errors

CMN Unlocks

MM Partition LU

Latitude

Longitude Name Description

63.99 -95.66 SB MM 21/11/2009

-46.72 -1.44 SB MM 22/11/2009

77.08 -159.44 SB MM 22/11/2009

-9.36 -57.58 SB MM 25/11/2009

-60.04 62.43 SB MM 30/11/2009

-0.70 133.00 SB MM 01/12/2009

54.00 -77.00 SB MM 05/12/2009

67.50 72.00SB

RAM 17/11/2009

51.00 -53.00SB

RAM 25/11/2009

21.27 -30.00SB

RAM 01/12/2009

-68.65 97.47 CMN C3 - 19/11/2009

-7.47 -27.20 CMN A2 - 19/11/2009

-62.39 102.39 CMN H1 - 24/11/2009

-9.20 -33.40 CMN A1 - 30/11/2009

-37.00 -43.00 CMN A2 - 01/12/2009

-78.00 -150.00 CMN C2 - 03/12/2009

-25.00 -11.00LATCH

UP 02/12/2009


MIRAS SODAP OPERATIONS: CMN UNLOCK

CMN Unlock: 6 cases observedThe ones in arm C show only Unlock flag, whereas those in Arm A (A1, A2, H1) have both Unlock and LO Out Power flagCharacterised by flag triggering, and sudden drop plus recovery of the PMS values.ARTS anomaly recorded SMOS-22No events since 03/12/09: 150 hours Guinness record!!!

CMN Date Time Lat Long

C3 19/11/2009 10:01:27.193 -68.65 97.47

A2 19/11/2009 19:44:29.555 -7.47 332.80

H1 24/11/2009 10:05:00.333 -62.39 102.39

A1 30/11/2009 08:21:21.518 -9.20 326.60

A2 01/12/2009 20:25:45.437 -37.00 317.00

C2 03/12/2009 00:57:27.234 -78.00 210.00


MIRAS SODAP OPERATIONS: CMN UNLOCK FOR A2

The lock after unlock is regained in all 6 cases.However, later analysis showed that A2 did not lock on the proper frequency in both cases.Analysis of frequency locking can only be done by correlation data processing, not through HKTM telemetry.After consultation with Project and Industry, the A2 recovery was defined through a switch off and on of complete arm A CNES intervention since switch is Proteus.Procedure defined PRO-CRP-300, validated in MIRASIM and EM.Switch off and on performed on the 01/12/09 at 13:50z.Later analysis showed that the Arm A started in default LICEF-U configuration, procedure updated to include a cycle calibration/dual post switch triggering.Open question about ARM switch qualified lifecyclesCurrent operational system procedure not formalised the A2 unlock requires later processing of data to ascertain proper locking (L1PP & SPQC changes in progress). Also procedure involves CNES operation, possibility to include it on current OIA as payload contingency.


MIRAS SODAP OPERATIONS: A2 RECOVERY

ARM A PMS

ARM B PMS

01/12/09 13:50


MIRAS SODAP OPERATIONS: CMN A2 DRIFT

The A2 LO Out Power flag has triggered 4 times in 3 distinct periods, always related to External Pointing Manoeuvres.Not seen in ECM on 08/12/09, after payload SW off and on 03/12/09 !!!In those cases, the LO Locking Flag has not triggered.This effect has not been observed in any of the other segment CMN’s.The PMS behaviour is not that of a sharp and sudden decrease, but rather a lowering of its value drifting away from the values of the adjacent segments in the armThe LO Out Flag duration is proportional to the manoeuvre duration.In the 3 cases recorded (1st, 2nd and 3rd of December) the drift away took place over the SAA region



CASE ON 01/12/09



CASE ON 02/12/09



CASE ON 02/12/09



ARM A SW Off

2nd A2 Unlock

2nd ECM3rd ECM

1st ECM PLM SW On





Last OCM

First A2 Unlock

No evidence found, though, of thermal correlation for the other CMN unlocks


MIRAS SODAP OPERATIONS: CMN A2 DRIFT 08/12/09


MIRAS SODAP OPERATIONS: PARTITION 7 LATCH UP

A partition 7 latch-up took place on the 02/12/00 at 06:59z

The effect is that the MM disables that partition. Reason for latch up is protection against over-current.

The geo-location of the event shows it over the SAA.

No associated CCU over-current detected

Recovered through payload SW OFF/ON on 03/12/09 at 15:40. Performed in one single pass, including later reprogramming of ITL CRP-SYS-100


MIRAS SODAP OPERATIONS: PARTITION 7 LATCH UP

A known SW problem prevents the high level SW (ASW) from proper managment of a reduced MM size. The LLSW is capable, though, of proper handling.

The impact is that X Band dumps that request a non valid new logical address are failed, and also that the write and read stop at (MM max size – 1 partition) not further progressing.

Ongoing tests in EM + analysis of possible SW change would be required. ARTS SMOS-24


MIRAS SODAP OPERATIONS: PARTITION 3 GAPS

MM scrubbing function has been reporting Double Bit errors since beginning of missionIn parallel, the SODAP data analysis has found data gaps in X Band data.Once analyzed, the gaps and the DB errors seem to be both related to a given MM partition, P3 (4th out of 12), and inside that partition, to an specific memory chip (7th), and even inside that chip, to an specific area

20 Gbits

1.6 Gbits

256 Mbits

~ 32 Mbits

~ 38xxxx to 4Bxxxx



Findings so far:Sometimes (not always), data is not written in some locations of the affected area, but the write pointer is updated.In those cases, the scrubbing function flags double bit errors when passing the first time after the write on the area, but not the following ones.The data in the non written areas is that of the last writing.The writing gap respects the science block, i.e. always 24 science packets are written before the gap. The gap duration is variable but less than one epoch. The proper data after gap starts with a reduced data block.



Scrubbing cycle is around 70 minutes.The scrubbing resolution is around 1520 MM packetsWriting over the region depends on instrument mode (around 35 hours in Full mode, and 60 in Dual)The gaps found in TM and the addresses reported in the scrubbing are not the same.The gaps are not consistent across different write passes on the area.Under investigation, one case of Partition 5 DB happened on the same inner chip and area, but no data gap found. P5 and P3 share the same slave FPGA controllerSome cases of DB EDAC have been triggered upon read operations over the area, but again not always.Problem reported in ARTS (SMOS-23)


MIRAS SODAP OPERATIONS: PARTITION 3 GAPSDOY Date Time Partition DB SB DB Adress HEX Device Address MM Packet

322 18/11/2009 10:00 P3 affected region crossing 1723794

322 18/11/2009 10:11:04 3 1 0 104623620 63C6E04 6 3C6E04 1718444

322 18/11/2009 10:11:08 3 1 0 105171971 644CC03 6 44CC03 1722130

324 20/11/2009 09:57:23 3 4 0 104995840 6421C00 6 421C00 1720946

324 20/11/2009 09:57:25 3 176 197 105265451 646392B 6 46392B 1722759

324 20/11/2009 11:07:12 3 99 23 105265401 64638F9 6 4638F9 1722759



324 20/11/2009 20:25:45 3 1 0 105598679 64B4ED7 6 4B4ED7 1724999


326 22/11/2009 07:19:29 3 1 0 104989734 6420426 6 420426 1720905

326 22/11/2009 07:19:30 3 11 1 105172050 644CC52 6 44CC52 1722131

326 22/11/2009 07:19:32 3 8 6 105263839 64632DF 6 4632DF 1722748


327 23/11/2009 19:35:51 3 158 37 104655616 63CEB00 6 3CEB00 1718659

329 25/11/2009 05:30:00 P3 affected region crossing NO GAP

330 26/11/2009 13:17:22 5 1 2 104300751 63780CF 6 3780CF 2618577

330 26/11/2009 17:00:00 P3 affected region crossing 1728192

330 26/11/2009 17:46:02 3 45 6 105266123 6463BCB 6 463BCB 1722763

332 28/11/2009 04:15 P3 affected region crossing NO GAP


333 29/11/2009 16:03:55 3 24 20 104380760 638B958 6 38B958 1716811

333 29/11/2009 16:03:58 3 5 4 104745930 63E4BCA 6 3E4BCA 1719266

335 01/12/2009 02:32 P3 affected region crossing


336 02/12/2009 14:12:35 3 73 14 105235711 645C4FF 6 45C4FF 1722559

336 02/12/2009 14:43:39 3 26 8 105174066 644D432 6 44D432 1722145

336 02/12/2009 14:43:41 3 47 6 105235711 645C4FF 6 45C4FF 1722559


338 04/12/2009 13:01:36 3 15 5 105080861 643681D 6 43681D 1721518

338 04/12/2009 13:01:36 3 33 22 105205195 6454DCB 6 454DCB 1722354


341 07/12/2009 08:33:05 3 33 9 105235659 645C4CB 6 45C4CB 1722559








FUTURE OPERATIONS & ANOMALY INVESTIGATION

MIRASIM + EM operational temperature reduction procedure validation in case it is needed.Definition of a system (i.e. DPGS + FOS + CNES) procedure to handle A2 recovery from wrong unlock.Further data gathering and characterization of P3 anomaly. However, detailed data analysis outside the scope of SODAP will be difficult.EM and MIRASIM validation of Latch up recovery without Power Off and On.Implementation of further limit monitoring and trend analysis.Required a better geo location capability for the different events, current AAV-PVT data not suitable. Workaround through SPGF too cumbersome.


ANOMALY SUMMARY

RAM single bit errors No action, monitoring

MM single bit errors No action, monitoring

C2 Nominal Thermostat (ARTS 21)

No action. Comment to be added in PRO-FCP-101CMN Unlock (ARTS 22)

Further investigations required. A2 unlock requires system procedure, ground tools and possibly other actions (i.e. cool down)

CMN A2 driftFurther investigations. Assessment of behaviour in slew locked in proper frequency.

Mass Memory Partition 6 (7th) Latch Up (ARTS 24)

Milder recovery recommended, further tests in EM. ASW modification required.

MM Partition 3 (4th) gap problems (ARTS 23)

Further investigations (industry is required). ASW modification and/or DPGS modification required.


CONCLUSIONS

No immediate stopper for ICOP blue phase startHowever, A2 CMN lock into wrong frequency would need characterization and operational fix if persistent.To be considered that current recovery procedure involves event detection, locking frequency definition out of correlation data, and recovery via Arm Power Cycle through Proteus, making use of the power switch required to assess its lifetime.Also, other possible SEU driven events (like MM P6 Latch Up) could reduce the data throughput and would require long engineering time for its fix (days of data loss). Partition failure under evaluation in the EM. SW change would alleviate the impact.Minor though systematic data loss in the P3 of Redundant MM. Under analysis for better characterization, still implies random data loss.The two previous events speak in favour of assessment of SW modification in order to handle gracefully MM partition disabling.


FOS CONFIGURATION

FOS CONFIGURATION


FOS CONFIGURATION: PLPC

Version 2.8Added one patch for better load of stacks in Manual StackTime Correlation settings changed, to cope with clock on board too fast, thus causing resets of function.Current workaround allows for maximum 10 msec deviation

Problems Re-ingestions store packets in different point in timeNot possible to cover a gap involving the switch off and on.Patch under production foreseen end of year.

Modification and cleanup in the Report generation, too many reports produced back to Gantt Explorer and SPGFModification of the verification window up to 3 weeks to cater for 2 weeks planning.Still pending verification with CNES of the TC REJECT files. To be exercised with validation chain.Agreement reached with CNES to receive during mission lifetime E_HKTM synchronized with the passSome data loss in S_Band data to be analyzed with CNES (punctual data gaps + G/S cases like HBK)


FOS CONFIGURATION: SPGF

3.3.1Some changes in the configuration

SVAL Auto commanding settings definedDefinition of nominal mode in order to expand the conflict analysis of external manoeuvres and zones

In use since 17/11/12 for the planningThe LO cyclic calibration introduces a long delay in the generation of the different products (~hours to run conflict analysis)The planning of external manoeuvres becomes more and more complex taking into consideration all the scientific constraints + S_Band visibilities + exclusion of zones. Could be even more difficult in case of avoidance of the P3 zone for calibrations.Under analysis a change to directly send requests to SVAL avoiding the DPGS Core.


FOS CONFIGURATION: PXMF & VALIDATION CHAIN

PXMF2.8Used during SODAP by SODAP teamData ingestion based on X_Band, non linear nor complete during SODAP.Kept up to date in DB and SW versionSystem proves useful in case of gaps in S_Band, however no consolidation of TM exercised yet.

VALIDATION CHAINMIRASIMPLPCDEVSPGF-2ESTEC MachinesAllow validation of DB’s, patches and procedures SPGF-2 used for geolocation


FOS CONFIGURATION: DATABASE & FOP

DataBase4.3 currently installedLaunch version was 4.0Main modifications are implementation of new limit monitoring.Implementation of common set of TM checks on Proteus TM together with CNES.

FOPImprovements to FCP-SDP-100Creation of new PRO-CRP-300 for ARM OFF/ON


FOS CONFIGURATION: MUST

Used for trend analysis and offline data analysisCurrently populated up to the best possible availability of dataManual population of Server DB tedious task, automatic population under analysis.The current MUST DB size grows too much for a complete mission (30 Gb for 1 month of effective mission)

Reduction of parametersIngestion of a selection of periods

Under analysis a replica of MUST external to the firewall for other users access (Industry + PLSO)Generation of Synthetic parameters in MUST ongoing


FOS CONFIGURATION: Web Server FTP

Public access of FOS related items available in:

193.146.123.163

/smos/FOS

Mission PlanningTxt tab separated + ESOV compatible formatsAvailable information for weekly scheduling including calibrations, manoeuvres and x_band visibilities.

AnomalyPlots and textual data of given events (i.e. CMN unlocks)

Operational Data InterchangeReports

In progress (this presentation available)

Others


FOS CONFIGURATION: STAFFING IN THE PERIOD

BS and GB fully on the job, extended shifts including some non working days.Amparo Garrigues still leading Mission PlanningStephen Cooke support in LEOP and SODAPMassimiliano Mazza System supportEADS Casa support teams for system and thermalRocio Fernandez in learning phase however heavily taken by DPGS activitiesESOC support (JP) + Elena Checa (ESTEC) for thermal

MANY THANKS TO ALL !!!

On Call service activeUnder definition the triggers for itNo triggering from CNES


FOS CONFIGURATION: Conclusion

FOS Ground Segment in really good shape to support rest of IOCP.Minimal anomalies detected, all of them corrected or ongoing.FOS configuration optimized after 5 weeks of operations, more alarms, better trend analysis support, refined mission planningAdequate manpower to support operations in this phaseOperational Interface Agreement definition ongoing with DPGSOn Call triggers being defined, service already available.

Pending:It is required a translation capability from Proteus AAV+PVT to Geo localization (BD contacted). Ideally to be implemented as Synthetic in PLPC, alternative would be in MUST.SPGF speed very much affected by high LO calibration frequency.PLPC Time Correlation problem to be fixed.Understanding of S_Band packet loss in certain passes.

Documents

IOCP Key Point 0 © Rafa Leon Guillermo Buenadicha OPS OEX SMOS LEOP & SODAP OPERATIONS © Rafa Leon