Result size still needs to be shrunk

[robobenklein]

[2020-02-20 04:59:31] [    INFO] [worker main_loop] Claimed job: 5e28c9271342c9331683f852, going to work.
[2020-02-20 04:59:31] [    INFO] [worker checkin] Checkin completed: setup
[2020-02-20 04:59:31] [    INFO] [worker checkin] Checkin completed: clone
[2020-02-20 04:59:31] [    INFO] [worker run_job] Beginning clone for job 5e28c9271342c9331683f852
[2020-02-20 04:59:35] [    INFO] [worker checkin] Checkin completed: analyze
[2020-02-20 05:01:29] [   ERROR] [worker run_job] Unknown error when submitting job result!
[2020-02-20 05:01:29] [ WARNING] [worker run_job] <Response [500]>
[2020-02-20 05:01:29] [   ERROR] [worker __main__] Unknown exception: Failed to communicate result to server.
[2020-02-20 05:01:29] [    INFO] [worker checkin] Checkin completed: error
Traceback (most recent call last):
  File "/home/ca-worker/venv/bin/ca-worker", line 8, in <module>
    sys.exit(__main__())
  File "/home/ca-worker/venv/lib/python3.6/site-packages/cadistributor/worker.py", line 257, in __main__
    raise e
  File "/home/ca-worker/venv/lib/python3.6/site-packages/cadistributor/worker.py", line 243, in __main__
    main_loop()
  File "/home/ca-worker/venv/lib/python3.6/site-packages/cadistributor/worker.py", line 271, in main_loop
    run_job(job)
  File "/home/ca-worker/venv/lib/python3.6/site-packages/cadistributor/worker.py", line 161, in run_job
    raise ConnectionError("Failed to communicate result to server.")
ConnectionError: Failed to communicate result to server.

because:

  File "/home/ca-jobserver/CodeAnalytics-distributor/venv/lib/python3.6/site-packages/pymongo/message.py", line 1077, in _raise_document_too_large
    raise DocumentTooLarge("%r command document too large" % (operation,))
pymongo.errors.DocumentTooLarge: 'update' command document too large
[2020-02-20 05:01:29] [   DEBUG] [app update_worker_status] worker coretester2 status: error 

worker state:

{
    "endpoint" : "https://ca-core.unhexium.dev",
    "error" : {
        "type" : "<class 'ConnectionError'>",
        "str" : "Failed to communicate result to server.",
        "time" : ISODate("2020-02-20T05:01:29.578Z")
    },
    "job" : {
        "_id" : ObjectId("5e28c9271342c9331683f852"),
        "function" : "caanalyzer.__main__:analyze",
        "repodir" : "job-5e28c9271342c9331683f852",
        "url" : "https://github.com/streamlit/streamlit.git",
        "workdir" : "/tmp/ca-workdir/ca-worker-coretester2"
    },
    "lastcheckin" : ISODate("2020-02-20T05:01:29.578Z"),
    "lastcheckin_human" : "2020-02-20 05:01:29 ",
    "status" : "error"
}

[argvrutter]

If we’re talking about json, its not outputting minified atm. In terms of object size reduction I think best outcome is just making it more configurable so that the dict only contains info you request.

…

On Thu, Feb 20, 2020 at 00:07 Ben Klein ***@***.***> wrote: Assigned #30 <#30> to @solsane <https://github.com/solsane>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#30?email_source=notifications&email_token=ACIXGYWZRTDN3VG3NTIC2VLRDYFY7A5CNFSM4KYHKF32YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOWYHDOUA#event-3054384976>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIXGYURXK7ICVIAS4BOUG3RDYFY7ANCNFSM4KYHKF3Q> .

[robobenklein]

Still a problem: pymongo.errors.DocumentTooLarge: BSON document too large (17069471 bytes) - the connected server supports BSON document sizes up to 16793598 bytes.

Ofc. a 17MB result is toooo big.

job https://github.com/demicoder/quanswer.git

[jball1997]

Wanna use GridFS?