spacy pretrain epco_loss

Hi @ines ,

I ran spacy pretrain. Its been running since 26 hours. Here is last the last 20 lines of log.console.

{"nr_word":740145943,"loss":208454483.9765625,"epoch_loss":2007579.0078125,"epoch":82}
{"nr_word":749063364,"loss":210458239.640625,"epoch_loss":2003755.6640625,"epoch":83}
{"nr_word":757980785,"loss":212457070.3046875,"epoch_loss":1998830.6640625,"epoch":84}
{"nr_word":766898206,"loss":214449781.5,"epoch_loss":1992711.1953125,"epoch":85}
{"nr_word":775815627,"loss":216440052.78125,"epoch_loss":1990271.28125,"epoch":86}
{"nr_word":784733048,"loss":218425152.15625,"epoch_loss":1985099.375,"epoch":87}
{"nr_word":793650469,"loss":220409435.125,"epoch_loss":1984282.96875,"epoch":88}
{"nr_word":802567890,"loss":222390616.5390625,"epoch_loss":1981181.4140625,"epoch":89}
{"nr_word":811485311,"loss":224367934.671875,"epoch_loss":1977318.1328125,"epoch":90}
{"nr_word":820402732,"loss":226339806.6875,"epoch_loss":1971872.015625,"epoch":91}
{"nr_word":829320153,"loss":228307787.3203125,"epoch_loss":1967980.6328125,"epoch":92}
{"nr_word":838237574,"loss":230273821.5234375,"epoch_loss":1966034.203125,"epoch":93}
{"nr_word":847154995,"loss":232239828.6875,"epoch_loss":1966007.1640625,"epoch":94}
{"nr_word":856072416,"loss":234200877.4296875,"epoch_loss":1961048.7421875,"epoch":95}
{"nr_word":864989837,"loss":236153965.28125,"epoch_loss":1953087.8515625,"epoch":96}
{"nr_word":873907258,"loss":238106919.0,"epoch_loss":1952953.71875,"epoch":97}
{"nr_word":882824679,"loss":240059697.6484375,"epoch_loss":1952778.6484375,"epoch":98}
{"nr_word":891742100,"loss":242002540.70703125,"epoch_loss":1942843.05859375,"epoch":99}
{"nr_word":900659521,"loss":243946234.09765625,"epoch_loss":1943693.390625,"epoch":100}
{"nr_word":909576942,"loss":245888442.37890625,"epoch_loss":1942208.28125,"epoch":101}

I am waiting for epoc_loss will be consistent. When do i stop that train?. does that seams good?.

Thanks,

loss is up and down. even after 250 epocs. I can wait for 1000 epocs but the problem is it needs to run ateleast 10 days to reach 1000 on 4 cores cpu not gpu machine.