Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] TensorDict logger #710

Merged
merged 3 commits into from
Mar 15, 2024
Merged

[Feature] TensorDict logger #710

merged 3 commits into from
Mar 15, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Mar 15, 2024

No description provided.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 15, 2024
@vmoens vmoens added the enhancement New feature or request label Mar 15, 2024
Copy link

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 127. Improved: $\large\color{#35bf28}6$. Worsened: $\large\color{#d91a1a}47$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_plain_set_nested 49.1020μs 22.8284μs 43.8050 KOps/s 66.3514 KOps/s $\textbf{\color{#d91a1a}-33.98\%}$
test_plain_set_stack_nested 58.5190μs 23.0974μs 43.2949 KOps/s 65.3457 KOps/s $\textbf{\color{#d91a1a}-33.74\%}$
test_plain_set_nested_inplace 84.6790μs 24.6517μs 40.5651 KOps/s 57.8405 KOps/s $\textbf{\color{#d91a1a}-29.87\%}$
test_plain_set_stack_nested_inplace 94.7470μs 24.8100μs 40.3063 KOps/s 57.8674 KOps/s $\textbf{\color{#d91a1a}-30.35\%}$
test_items 29.7650μs 2.3962μs 417.3310 KOps/s 388.8413 KOps/s $\textbf{\color{#35bf28}+7.33\%}$
test_items_nested 0.8997ms 0.2722ms 3.6741 KOps/s 3.7327 KOps/s $\color{#d91a1a}-1.57\%$
test_items_nested_locked 0.3513ms 0.2724ms 3.6715 KOps/s 3.6981 KOps/s $\color{#d91a1a}-0.72\%$
test_items_nested_leaf 0.7735ms 0.1728ms 5.7863 KOps/s 6.0268 KOps/s $\color{#d91a1a}-3.99\%$
test_items_stack_nested 0.3478ms 0.2752ms 3.6331 KOps/s 3.7013 KOps/s $\color{#d91a1a}-1.84\%$
test_items_stack_nested_leaf 0.3060ms 0.1727ms 5.7904 KOps/s 6.0339 KOps/s $\color{#d91a1a}-4.03\%$
test_items_stack_nested_locked 0.9887ms 0.2763ms 3.6188 KOps/s 3.6733 KOps/s $\color{#d91a1a}-1.48\%$
test_keys 24.1450μs 3.9080μs 255.8831 KOps/s 250.6006 KOps/s $\color{#35bf28}+2.11\%$
test_keys_nested 2.3268ms 0.1598ms 6.2581 KOps/s 6.9036 KOps/s $\textbf{\color{#d91a1a}-9.35\%}$
test_keys_nested_locked 0.3404ms 0.1634ms 6.1201 KOps/s 6.7339 KOps/s $\textbf{\color{#d91a1a}-9.11\%}$
test_keys_nested_leaf 39.7129ms 0.1484ms 6.7391 KOps/s 8.0236 KOps/s $\textbf{\color{#d91a1a}-16.01\%}$
test_keys_stack_nested 0.2978ms 0.1619ms 6.1758 KOps/s 6.7240 KOps/s $\textbf{\color{#d91a1a}-8.15\%}$
test_keys_stack_nested_leaf 0.2247ms 0.1419ms 7.0470 KOps/s 7.8261 KOps/s $\textbf{\color{#d91a1a}-9.96\%}$
test_keys_stack_nested_locked 0.3503ms 0.1653ms 6.0488 KOps/s 6.5310 KOps/s $\textbf{\color{#d91a1a}-7.38\%}$
test_values 11.0757μs 1.1591μs 862.7467 KOps/s 865.8357 KOps/s $\color{#d91a1a}-0.36\%$
test_values_nested 0.1171ms 57.7723μs 17.3093 KOps/s 18.9068 KOps/s $\textbf{\color{#d91a1a}-8.45\%}$
test_values_nested_locked 0.1161ms 57.7046μs 17.3296 KOps/s 19.2667 KOps/s $\textbf{\color{#d91a1a}-10.05\%}$
test_values_nested_leaf 0.1201ms 53.1170μs 18.8264 KOps/s 21.4280 KOps/s $\textbf{\color{#d91a1a}-12.14\%}$
test_values_stack_nested 94.0350μs 58.4140μs 17.1192 KOps/s 18.7524 KOps/s $\textbf{\color{#d91a1a}-8.71\%}$
test_values_stack_nested_leaf 0.1074ms 52.6862μs 18.9803 KOps/s 21.5939 KOps/s $\textbf{\color{#d91a1a}-12.10\%}$
test_values_stack_nested_locked 0.1351ms 58.6213μs 17.0586 KOps/s 18.8348 KOps/s $\textbf{\color{#d91a1a}-9.43\%}$
test_membership 43.7820μs 1.3372μs 747.8356 KOps/s 736.2343 KOps/s $\color{#35bf28}+1.58\%$
test_membership_nested 21.3600μs 3.4428μs 290.4603 KOps/s 289.7430 KOps/s $\color{#35bf28}+0.25\%$
test_membership_nested_leaf 51.0060μs 3.4284μs 291.6845 KOps/s 291.6127 KOps/s $\color{#35bf28}+0.02\%$
test_membership_stacked_nested 33.1820μs 3.4193μs 292.4561 KOps/s 296.2858 KOps/s $\color{#d91a1a}-1.29\%$
test_membership_stacked_nested_leaf 20.4380μs 3.4511μs 289.7615 KOps/s 295.9305 KOps/s $\color{#d91a1a}-2.08\%$
test_membership_nested_last 57.9680μs 4.3299μs 230.9536 KOps/s 238.5276 KOps/s $\color{#d91a1a}-3.18\%$
test_membership_nested_leaf_last 26.6000μs 4.3217μs 231.3928 KOps/s 236.2106 KOps/s $\color{#d91a1a}-2.04\%$
test_membership_stacked_nested_last 44.8340μs 4.2420μs 235.7403 KOps/s 188.2248 KOps/s $\textbf{\color{#35bf28}+25.24\%}$
test_membership_stacked_nested_leaf_last 22.7630μs 4.2651μs 234.4630 KOps/s 187.0103 KOps/s $\textbf{\color{#35bf28}+25.37\%}$
test_nested_getleaf 56.2350μs 11.0043μs 90.8735 KOps/s 93.5802 KOps/s $\color{#d91a1a}-2.89\%$
test_nested_get 54.8520μs 10.4831μs 95.3913 KOps/s 97.5272 KOps/s $\color{#d91a1a}-2.19\%$
test_stacked_getleaf 77.3650μs 10.6560μs 93.8440 KOps/s 93.3554 KOps/s $\color{#35bf28}+0.52\%$
test_stacked_get 54.8320μs 10.0026μs 99.9738 KOps/s 98.6833 KOps/s $\color{#35bf28}+1.31\%$
test_nested_getitemleaf 56.6860μs 11.2660μs 88.7625 KOps/s 88.4374 KOps/s $\color{#35bf28}+0.37\%$
test_nested_getitem 52.5880μs 10.4194μs 95.9747 KOps/s 95.9771 KOps/s $-0.00\%$
test_stacked_getitemleaf 35.5360μs 11.3784μs 87.8860 KOps/s 87.8265 KOps/s $\color{#35bf28}+0.07\%$
test_stacked_getitem 53.7500μs 10.3106μs 96.9877 KOps/s 96.6027 KOps/s $\color{#35bf28}+0.40\%$
test_lock_nested 1.1638ms 0.3499ms 2.8576 KOps/s 2.8909 KOps/s $\color{#d91a1a}-1.15\%$
test_lock_stack_nested 0.6107ms 0.3099ms 3.2268 KOps/s 3.3948 KOps/s $\color{#d91a1a}-4.95\%$
test_unlock_nested 99.9742ms 0.4511ms 2.2168 KOps/s 2.2631 KOps/s $\color{#d91a1a}-2.05\%$
test_unlock_stack_nested 0.5152ms 0.3182ms 3.1431 KOps/s 3.2844 KOps/s $\color{#d91a1a}-4.30\%$
test_flatten_speed 0.6455ms 0.2809ms 3.5600 KOps/s 3.7231 KOps/s $\color{#d91a1a}-4.38\%$
test_unflatten_speed 0.6186ms 0.4464ms 2.2402 KOps/s 2.4702 KOps/s $\textbf{\color{#d91a1a}-9.31\%}$
test_common_ops 4.4494ms 0.7383ms 1.3544 KOps/s 1.5442 KOps/s $\textbf{\color{#d91a1a}-12.29\%}$
test_creation 21.0400μs 1.8689μs 535.0750 KOps/s 546.9107 KOps/s $\color{#d91a1a}-2.16\%$
test_creation_empty 58.5100μs 11.7866μs 84.8420 KOps/s 129.5713 KOps/s $\textbf{\color{#d91a1a}-34.52\%}$
test_creation_nested_1 41.0670μs 14.3293μs 69.7872 KOps/s 97.9058 KOps/s $\textbf{\color{#d91a1a}-28.72\%}$
test_creation_nested_2 0.1056ms 18.3465μs 54.5063 KOps/s 74.5374 KOps/s $\textbf{\color{#d91a1a}-26.87\%}$
test_clone 77.5550μs 13.7656μs 72.6446 KOps/s 75.3740 KOps/s $\color{#d91a1a}-3.62\%$
test_getitem[int] 32.1200μs 11.4573μs 87.2807 KOps/s 91.1931 KOps/s $\color{#d91a1a}-4.29\%$
test_getitem[slice_int] 85.0890μs 24.0556μs 41.5704 KOps/s 45.1611 KOps/s $\textbf{\color{#d91a1a}-7.95\%}$
test_getitem[range] 0.1448ms 42.6030μs 23.4725 KOps/s 24.5131 KOps/s $\color{#d91a1a}-4.24\%$
test_getitem[tuple] 49.9430μs 18.9200μs 52.8540 KOps/s 55.4813 KOps/s $\color{#d91a1a}-4.74\%$
test_getitem[list] 0.2910ms 37.3046μs 26.8063 KOps/s 27.4892 KOps/s $\color{#d91a1a}-2.48\%$
test_setitem_dim[int] 69.4600μs 36.3180μs 27.5345 KOps/s 32.8246 KOps/s $\textbf{\color{#d91a1a}-16.12\%}$
test_setitem_dim[slice_int] 0.1238ms 64.9308μs 15.4010 KOps/s 17.8514 KOps/s $\textbf{\color{#d91a1a}-13.73\%}$
test_setitem_dim[range] 0.1463ms 82.0835μs 12.1827 KOps/s 13.7514 KOps/s $\textbf{\color{#d91a1a}-11.41\%}$
test_setitem_dim[tuple] 97.4420μs 52.1589μs 19.1722 KOps/s 21.4600 KOps/s $\textbf{\color{#d91a1a}-10.66\%}$
test_setitem 95.9300μs 21.4191μs 46.6872 KOps/s 53.3345 KOps/s $\textbf{\color{#d91a1a}-12.46\%}$
test_set 0.1088ms 20.8143μs 48.0439 KOps/s 55.5916 KOps/s $\textbf{\color{#d91a1a}-13.58\%}$
test_set_shared 1.5699ms 0.1422ms 7.0300 KOps/s 6.8168 KOps/s $\color{#35bf28}+3.13\%$
test_update 0.1426ms 23.9058μs 41.8308 KOps/s 50.3798 KOps/s $\textbf{\color{#d91a1a}-16.97\%}$
test_update_nested 0.1672ms 32.9967μs 30.3060 KOps/s 35.4101 KOps/s $\textbf{\color{#d91a1a}-14.41\%}$
test_update__nested 0.1285ms 25.5855μs 39.0847 KOps/s 39.5736 KOps/s $\color{#d91a1a}-1.24\%$
test_set_nested 99.8970μs 23.2415μs 43.0266 KOps/s 49.8863 KOps/s $\textbf{\color{#d91a1a}-13.75\%}$
test_set_nested_new 0.1559ms 26.4738μs 37.7732 KOps/s 40.8402 KOps/s $\textbf{\color{#d91a1a}-7.51\%}$
test_select 0.1072ms 41.8789μs 23.8784 KOps/s 25.8169 KOps/s $\textbf{\color{#d91a1a}-7.51\%}$
test_select_nested 0.1609ms 65.4170μs 15.2865 KOps/s 16.7124 KOps/s $\textbf{\color{#d91a1a}-8.53\%}$
test_exclude_nested 0.1857ms 0.1248ms 8.0110 KOps/s 7.8799 KOps/s $\color{#35bf28}+1.66\%$
test_empty[True] 0.7608ms 0.4237ms 2.3601 KOps/s 2.4203 KOps/s $\color{#d91a1a}-2.49\%$
test_empty[False] 8.6864μs 1.0402μs 961.3706 KOps/s 955.5154 KOps/s $\color{#35bf28}+0.61\%$
test_unbind_speed 0.4574ms 0.2544ms 3.9311 KOps/s 4.0296 KOps/s $\color{#d91a1a}-2.45\%$
test_unbind_speed_stack0 0.3921ms 0.2476ms 4.0380 KOps/s 4.2083 KOps/s $\color{#d91a1a}-4.05\%$
test_unbind_speed_stack1 0.1312s 0.6903ms 1.4487 KOps/s 1.4726 KOps/s $\color{#d91a1a}-1.62\%$
test_split 0.1275s 1.7386ms 575.1826 Ops/s 599.0589 Ops/s $\color{#d91a1a}-3.99\%$
test_chunk 2.5466ms 1.5286ms 654.2144 Ops/s 685.7058 Ops/s $\color{#d91a1a}-4.59\%$
test_creation[device0] 0.1927ms 0.1016ms 9.8403 KOps/s 9.6789 KOps/s $\color{#35bf28}+1.67\%$
test_creation_from_tensor 5.8907ms 87.3518μs 11.4480 KOps/s 12.0835 KOps/s $\textbf{\color{#d91a1a}-5.26\%}$
test_add_one[memmap_tensor0] 91.8420μs 5.5381μs 180.5679 KOps/s 177.2005 KOps/s $\color{#35bf28}+1.90\%$
test_contiguous[memmap_tensor0] 11.9920μs 0.6359μs 1.5726 MOps/s 1.6076 MOps/s $\color{#d91a1a}-2.18\%$
test_stack[memmap_tensor0] 22.6630μs 3.7556μs 266.2722 KOps/s 280.5027 KOps/s $\textbf{\color{#d91a1a}-5.07\%}$
test_memmaptd_index 1.0775ms 0.2381ms 4.2008 KOps/s 4.0530 KOps/s $\color{#35bf28}+3.65\%$
test_memmaptd_index_astensor 0.5326ms 0.3095ms 3.2306 KOps/s 3.2775 KOps/s $\color{#d91a1a}-1.43\%$
test_memmaptd_index_op 1.3279ms 0.6461ms 1.5478 KOps/s 1.7382 KOps/s $\textbf{\color{#d91a1a}-10.96\%}$
test_serialize_model 0.2248s 0.1149s 8.7008 Ops/s 8.3782 Ops/s $\color{#35bf28}+3.85\%$
test_serialize_model_pickle 0.4659s 0.3750s 2.6663 Ops/s 2.6039 Ops/s $\color{#35bf28}+2.40\%$
test_serialize_weights 0.1006s 96.2971ms 10.3845 Ops/s 9.8036 Ops/s $\textbf{\color{#35bf28}+5.93\%}$
test_serialize_weights_returnearly 0.1260s 0.1204s 8.3086 Ops/s 7.9031 Ops/s $\textbf{\color{#35bf28}+5.13\%}$
test_serialize_weights_pickle 0.7904s 0.5388s 1.8559 Ops/s 2.4817 Ops/s $\textbf{\color{#d91a1a}-25.22\%}$
test_serialize_weights_filesystem 0.1021s 92.2087ms 10.8450 Ops/s 10.0875 Ops/s $\textbf{\color{#35bf28}+7.51\%}$
test_serialize_model_filesystem 0.1013s 96.4242ms 10.3708 Ops/s 10.3900 Ops/s $\color{#d91a1a}-0.18\%$
test_reshape_pytree 60.6230μs 20.7225μs 48.2567 KOps/s 47.0727 KOps/s $\color{#35bf28}+2.52\%$
test_reshape_td 90.0390μs 33.6882μs 29.6840 KOps/s 31.3141 KOps/s $\textbf{\color{#d91a1a}-5.21\%}$
test_view_pytree 55.7740μs 20.4599μs 48.8760 KOps/s 47.7547 KOps/s $\color{#35bf28}+2.35\%$
test_view_td 0.1446s 65.3829μs 15.2945 KOps/s 15.6115 KOps/s $\color{#d91a1a}-2.03\%$
test_unbind_pytree 68.1880μs 24.5900μs 40.6669 KOps/s 40.9696 KOps/s $\color{#d91a1a}-0.74\%$
test_unbind_td 0.1414ms 38.1906μs 26.1845 KOps/s 27.7111 KOps/s $\textbf{\color{#d91a1a}-5.51\%}$
test_split_pytree 61.4050μs 24.4116μs 40.9641 KOps/s 41.9119 KOps/s $\color{#d91a1a}-2.26\%$
test_split_td 0.1253ms 42.3031μs 23.6389 KOps/s 25.5798 KOps/s $\textbf{\color{#d91a1a}-7.59\%}$
test_add_pytree 88.9370μs 30.8469μs 32.4181 KOps/s 33.1259 KOps/s $\color{#d91a1a}-2.14\%$
test_add_td 0.1512ms 61.2593μs 16.3240 KOps/s 20.0207 KOps/s $\textbf{\color{#d91a1a}-18.46\%}$
test_distributed 0.1888ms 99.2671μs 10.0738 KOps/s 9.7375 KOps/s $\color{#35bf28}+3.45\%$
test_tdmodule 49.1020μs 19.1213μs 52.2976 KOps/s 62.9285 KOps/s $\textbf{\color{#d91a1a}-16.89\%}$
test_tdmodule_dispatch 78.2660μs 36.9757μs 27.0448 KOps/s 32.0912 KOps/s $\textbf{\color{#d91a1a}-15.73\%}$
test_tdseq 41.5270μs 21.8917μs 45.6794 KOps/s 54.5819 KOps/s $\textbf{\color{#d91a1a}-16.31\%}$
test_tdseq_dispatch 70.7630μs 41.3820μs 24.1651 KOps/s 28.2128 KOps/s $\textbf{\color{#d91a1a}-14.35\%}$
test_instantiation_functorch 1.5765ms 1.3081ms 764.4803 Ops/s 749.1818 Ops/s $\color{#35bf28}+2.04\%$
test_instantiation_td 1.4784ms 1.0067ms 993.3389 Ops/s 971.0352 Ops/s $\color{#35bf28}+2.30\%$
test_exec_functorch 0.2975ms 0.1590ms 6.2910 KOps/s 6.4375 KOps/s $\color{#d91a1a}-2.28\%$
test_exec_functional_call 0.2920ms 0.1465ms 6.8242 KOps/s 6.8353 KOps/s $\color{#d91a1a}-0.16\%$
test_exec_td 0.2674ms 0.1423ms 7.0271 KOps/s 7.1581 KOps/s $\color{#d91a1a}-1.83\%$
test_exec_td_decorator 0.7641ms 0.2004ms 4.9898 KOps/s 5.1704 KOps/s $\color{#d91a1a}-3.49\%$
test_vmap_mlp_speed[True-True] 0.6421ms 0.4798ms 2.0842 KOps/s 2.1618 KOps/s $\color{#d91a1a}-3.59\%$
test_vmap_mlp_speed[True-False] 0.7492ms 0.4740ms 2.1096 KOps/s 2.1567 KOps/s $\color{#d91a1a}-2.18\%$
test_vmap_mlp_speed[False-True] 0.5629ms 0.3834ms 2.6082 KOps/s 2.5951 KOps/s $\color{#35bf28}+0.50\%$
test_vmap_mlp_speed[False-False] 0.6189ms 0.3833ms 2.6089 KOps/s 2.5955 KOps/s $\color{#35bf28}+0.51\%$
test_vmap_mlp_speed_decorator[True-True] 1.1924ms 0.5053ms 1.9792 KOps/s 2.0391 KOps/s $\color{#d91a1a}-2.94\%$
test_vmap_mlp_speed_decorator[True-False] 0.8167ms 0.5023ms 1.9910 KOps/s 2.0724 KOps/s $\color{#d91a1a}-3.93\%$
test_vmap_mlp_speed_decorator[False-True] 0.7358ms 0.4070ms 2.4573 KOps/s 2.4986 KOps/s $\color{#d91a1a}-1.65\%$
test_vmap_mlp_speed_decorator[False-False] 0.6670ms 0.4050ms 2.4690 KOps/s 2.4955 KOps/s $\color{#d91a1a}-1.06\%$
test_to_module_speed[True] 1.6879ms 1.4584ms 685.7014 Ops/s 726.6085 Ops/s $\textbf{\color{#d91a1a}-5.63\%}$
test_to_module_speed[False] 2.9174ms 1.4278ms 700.3993 Ops/s 727.5884 Ops/s $\color{#d91a1a}-3.74\%$

@vmoens vmoens merged commit f1c833e into main Mar 15, 2024
45 of 48 checks passed
@vmoens vmoens deleted the logger branch March 15, 2024 11:55
vmoens added a commit that referenced this pull request Mar 25, 2024
(cherry picked from commit f1c833e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants