FOSDIC Deck 195 Decoding¶

digital collection of FOSDIC data can be obtained at https://rda.ucar.edu/datasets/ds258.4/index.html#!sfol-wl-/data/ds258.4
pdf explaining original deck 195 format can be found at https://icoads.noaa.gov/reclaim/pdf/

Decoding Details¶

cassette / reel information is maintained in headerlines that begin with ‘00000195’, other details about the FOSDIC conversion format not pertaining to the original records are available at https://rda.ucar.edu/datasets/ds258.4/docs/fosdic_description.txt
This project is focused on providing ship type information and connecting to meta information for SST/Seawter Temp Bias (but also to look into air bias). To that end, present weather and sea state will remain as strings but ship number and temperatures will get converted per description.
Final entry will enclude the complete FOSDIC record entry for future reference
Location, date/time will also be converted to geocoordinates.
The only QC that will happen other than conversion checks, is to remove empty entries
formatting (decimals and integers) should remain close to initial values if possible, however, replacing invalid data with NaN (pythonic) is a float operation… the raw character string is made available for the purposes of addressing this type of conversion challenge.

import pandas as pd
import numpy as np

import datetime

with open('../../FOSDIC_COPY/fosdic_cd195','r') as f:
    file = f.readlines()

#define some essential conversion functions from the Deck 195 Manual

def wind_dir(x):
    try:
        return int(int(x)*10)
    except:
        return np.nan

def slp(x):
    try:
        return float(x)/100
    except:
        return np.nan
    
def airtemp(x):
    if x[0] == '0':
        try:
            return (float(x))
        except:
            return np.nan
    elif x[0] != ' ':
        try:
            return (-1*float(x[1::]))
        except:
            return np.nan
    else:
        return np.nan
    
def watertemp(x):
        try:
            return (float(x))
        except:
            return np.nan    

def inport(x): #make anything other than blank "TRUE"
    if x[0] != ' ':  
        return True
    else:
        return False

def geo_loc(quadrant,latitude,longitude):
    try:
        if quadrant == '0':
            return [int(latitude),-1*int(longitude)]
        elif quadrant == '1':
            return [int(latitude),int(longitude)]    
        elif quadrant == '2':
            return [-1*int(latitude),-1*int(longitude)]
        elif quadrant == '3':
            return [-1*int(latitude),int(longitude)]
        else:
            return [np.nan,np.nan]
    except:
            return [np.nan,np.nan]
        

counter = 0
data = {}
for count, row in enumerate(file):
    if (row[:8] == '00000195'):
        print(f'Headerlines: rowumber-{count}')
    elif row == '                                                                                \n': #blank line
        continue
    else:
        #create dictionary where key is row number                    
        data.update({counter:{'shipclass':row[0:2],
                    'shipno':row[2:5],
                    'year':row[5:7],
                    'month':row[7:9],
                    'day':row[9:11],
                    'hour':row[11:13],
                    'quadrant':row[13:14],
                    'lat_coded':row[14:16],
                    'lon_coded':row[16:19],
                    'ship_speed_kts':row[19:21], #knots
                    'ship_course_deg':row[21:23], #degrees
                    'wind_dir_deg':wind_dir(row[23:25]), #*10 as its cut to nearest ten degrees - and make integer, not float
                    'wind_speed_kts':row[25:27], #knots
                    'sealevelpressure_inHg':slp(row[27:31]), #in
                    'drybulb_temperature_degF':airtemp(row[31:34]), #degF
                    'wetbulb_temperature_degF':airtemp(row[34:37]), #degF
                    'water_injection_temperature_degF':watertemp(row[37:39]), #degF
                    'present_wx':row[39:48], #see encoding but this involves multiple fields: wx, clouds, vis
                    'sea_surface_temperature_degF':watertemp(row[48:50]), #degF
                    'sea_and_swell':row[50:56], #see encoding but this involves multiple fields
                    'inport_obs_indicator':inport(row[56:57]), #X=True,blank=False
                    #58 on - not used
                    'latitude_DegN':geo_loc(row[13:14],row[14:16],row[16:19])[0], #+N,+E
                    'longitude_DegE':geo_loc(row[13:14],row[14:16],row[16:19])[1], #+N,+E
                    'datetime':'19'+row[5:7]+'-'+row[7:9]+'-'+row[9:11]+' '+row[11:13]+':00:00',
                    'rawentry_80char':row
                    }})
        counter +=1

        

Headerlines: rowumber-0
Headerlines: rowumber-11900
Headerlines: rowumber-24532
Headerlines: rowumber-36153
Headerlines: rowumber-48354
Headerlines: rowumber-60146
Headerlines: rowumber-72265
Headerlines: rowumber-84308
Headerlines: rowumber-96273
Headerlines: rowumber-108434
Headerlines: rowumber-120708
Headerlines: rowumber-133038
Headerlines: rowumber-145374
Headerlines: rowumber-157128
Headerlines: rowumber-169710
Headerlines: rowumber-181385
Headerlines: rowumber-193689
Headerlines: rowumber-205917
Headerlines: rowumber-217943
Headerlines: rowumber-229640
Headerlines: rowumber-241251
Headerlines: rowumber-252753
Headerlines: rowumber-264969
Headerlines: rowumber-277353
Headerlines: rowumber-289685
Headerlines: rowumber-301329
Headerlines: rowumber-313251
Headerlines: rowumber-325285
Headerlines: rowumber-337365
Headerlines: rowumber-348882
Headerlines: rowumber-360892
Headerlines: rowumber-372911
Headerlines: rowumber-385279
Headerlines: rowumber-397246
Headerlines: rowumber-409214
Headerlines: rowumber-421541
Headerlines: rowumber-433915
Headerlines: rowumber-445552
Headerlines: rowumber-457497
Headerlines: rowumber-470070
Headerlines: rowumber-481914
Headerlines: rowumber-493433
Headerlines: rowumber-505361
Headerlines: rowumber-517759
Headerlines: rowumber-529326
Headerlines: rowumber-541973
Headerlines: rowumber-553944
Headerlines: rowumber-565653
Headerlines: rowumber-577107
Headerlines: rowumber-589695
Headerlines: rowumber-601358
Headerlines: rowumber-613256
Headerlines: rowumber-625754
Headerlines: rowumber-636789

df = pd.DataFrame.from_dict(data,orient='index')

df.to_csv('../data/FOSDIC_cd195.csv',index=False)

EcoSystems Fisheries-Oceanograpy Coordinated Investigations (EcoFOCI)

FOSDIC Deck 195 Decoding¶

Decoding Details¶

Some Initial Stats¶

Prepare for Merging with Vessel Meta Archive¶