?? fixs recovery data in api/v2 jhu #316

codedawi · 2020-05-20T03:19:01Z

reimplements the get_category("recovered") methods in jhu service.
adds/fixes the tests for recovered data now.
resolves the merge issue related to missing recovery data from select countries.
- Sierra Leone
- Netherlands, Bonaire, Sint Eustatius and Saba
- Malawi
- France, Saint Pierre and Miquelon
- United Kingdom, Falkland Islands (Malvinas)
- South Sudan
- Western Sahara
- Sao Tome and Principe
- Yemen
- Comoros
- Tajikistan
- Lesotho

While reviewing the jhu service code I uncovered at critical bug that could expose this service to data & uptime issues in the future. In the PR the function parse_history is implemented as a temporary fix which will prevent the service from going down and providing consumers inaccurate data.

Add note to code:

# ***************************************************************************

# TODO: This iteration approach assumes the indexes remain the same

#       and opens us to a CRITICAL ERROR. The removal of a column in the data source

#       would break the API or SHIFT all the data confirmed, deaths, recovery producting

#       incorrect data to consumers.

# ***************************************************************************

Potential address in @Kilo59 's refactor mentioned.

FIXES : #161 #276 #113

FIXES : codedawi/covid19-badges#1

* formatting (120 length -> 100) * csbs to/from Redis * nyt to/from Redis * ignore locustfile * unused coordinates

* add sentry-asgi * sentry middleware

Kilo59 · 2020-05-20T20:15:25Z

@codedawi Hey thanks for this! ?? I'll take a look at this ASAP.

Kilo59 · 2020-05-20T21:13:00Z

@codedawi It definitely appears to have fixed the recoveries of many locations (US, AU, etc), but many other locations that are not showing up in the logs as having a "merge error" (DE, CN, CA etc) still show zero recoveries (for me locally).

I'm almost inclined to merge it anyways ?? .
However certain locations such as the UK are showing partial recovery numbers instead, which IMO is worse than still showing 0.

Any API users please feel free to chime in.

{
  
"latest"
: {
    
"confirmed"
: 
250138
,
    
"deaths"
: 
35422
,
    
"recovered"
: 
1067

  },
  
"locations"
: [
    {
      
"id"
: 
217
,
      
"country"
: 
"
United Kingdom
"
,
      
"country_code"
: 
"
GB
"
,
      
"country_population"
: 
66488991
,
      
"province"
: 
"
Bermuda
"
,
      
"last_updated"
: 
"
2020-05-20T20:52:14.791765Z
"
,
      
"coordinates"
: {
        
"latitude"
: 
"
32.3078
"
,
        
"longitude"
: 
"
-64.7505
"

      },
      
"latest"
: {
        
"confirmed"
: 
125
,
        
"deaths"
: 
9
,
        
"recovered"
: 
78

      }
    },
...

    {
      
"id"
: 
223
,
      
"country"
: 
"
United Kingdom
"
,
      
"country_code"
: 
"
GB
"
,
      
"country_population"
: 
66488991
,
      
"province"
: 
"
"
,
      
"last_updated"
: 
"
2020-05-20T20:52:14.806740Z
"
,
      
"coordinates"
: {
        
"latitude"
: 
"
55.3781
"
,
        
"longitude"
: 
"
-3.436
"

      },
      
"latest"
: {
        
"confirmed"
: 
248818
,
        
"deaths"
: 
35341
,
        
"recovered"
: 
0

      }
    },
...

    {
      
"id"
: 
257
,
      
"country"
: 
"
United Kingdom
"
,
      
"country_code"
: 
"
GB
"
,
      
"country_population"
: 
66488991
,
      
"province"
: 
"
Falkland Islands (Malvinas)
"
,
      
"last_updated"
: 
"
2020-05-20T20:52:14.884740Z
"
,
      
"coordinates"
: {
        
"latitude"
: 
"
-51.7963
"
,
        
"longitude"
: 
"
-59.5236
"

      },
      
"latest"
: {
        
"confirmed"
: 
13
,
        
"deaths"
: 
0
,
        
"recovered"
: 
0

      }
    }
  ]
}

I think I would like to merge this immediately to the US hosted version of the API that's based off my own fork .

codedawi · 2020-08-21T21:03:49Z

Any update on getting this merged in? If not we can just close.

Kilo59 and others added 15 commits May 3, 2020 16:50

reduce cache size ( ExpDev07#21 )

6e5bf9f

Update Procfile

da822ba

Update Procfile

b085652

Update jhu.py

a1dcfb4

update max requests and max jitters

cc768a9

Add .deepsource.toml

ba056ce

Update Procfile

e54724e

Cache mem optimize ( ExpDev07#23 )

170eb12

* formatting (120 length -> 100) * csbs to/from Redis * nyt to/from Redis * ignore locustfile * unused coordinates

Sentry ( ExpDev07#24 )

7fe38b9

* add sentry-asgi * sentry middleware

use official middleware ( ExpDev07#25 )

96efc92

cache redis json serialization error

ecb0a68

fix nyt redis serialization error

464453d

?? fixing recovery data in api/v2 jhu

52ef1cd

?? fix linting issue

4106b7c

? unittest for parse val history

761f62d

codedawi force-pushed the master branch 2 times, most recently from 6a0bea5 to 761f62d Compare May 20, 2020 05:18

This was linked to issues May 20, 2020

Returns 0 recovered #237

Closed

Recovered time series are gone #200

Open

Recovered values is 0 on all request types. #242

Closed

JHU back with recovered cases. #276

Open

Kilo59 self-assigned this May 20, 2020

Kilo59 added source: jhu help wanted Extra attention is needed labels May 20, 2020

Kilo59 mentioned this pull request May 20, 2020

partial recovery fix Kilo59/coronavirus-tracker-api#31

Merged

Merge branch 'master' into master

0f3009e

codedawi closed this Dec 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

?? fixs recovery data in api/v2 jhu #316

?? fixs recovery data in api/v2 jhu #316

codedawi commented May 20, 2020 •

edited

Kilo59 commented May 20, 2020

Kilo59 commented May 20, 2020 •

edited

codedawi commented Aug 21, 2020

?? fixs recovery data in api/v2 jhu #316

?? fixs recovery data in api/v2 jhu #316

Conversation

codedawi commented May 20, 2020 • edited

Kilo59 commented May 20, 2020

Kilo59 commented May 20, 2020 • edited

codedawi commented Aug 21, 2020

codedawi commented May 20, 2020 •

edited

Kilo59 commented May 20, 2020 •

edited