ROADMAP will simplify this process in the next version - using a reporting web page
- Login to the server
- Place yourself at the working directory - i.e. /opt/caraone_data1 for example
- Run this command to make reporting where all the tickets are much simpler:
find . -maxdepth 1 -type d -exec sh -c 'echo -n "{} : "; find "{}" -type f | wc -l' \; | sort -n - Consider this command running as a job that updates every "X" seconds/minutes. Don't forget to be at the /opt/caraone_data1 level. It will output the numbers in each of the jobs_analyze_media and jobs_done directories for easier reading:
while systemctl is-active --quiet runner; do echo " "
echo " <<<<<<<<<< Jobs_done ($(date '+%Y-%m-%d %H:%M:%S')) >>>>>>>>>>"; echo "video_: $(find jobs_done/ -maxdepth 1 -name 'video_*' | wc -l), picture_: $(find jobs_done/ -maxdepth 1 -name 'picture_*' | wc -l), pdf_: $(find jobs_done/ -maxdepth 1 -name 'pdf_*' | wc -l), txt_: $(find jobs_done/ -maxdepth 1 -name 'txt_*' | wc -l), avb_: $(find jobs_done/ -maxdepth 1 -name 'avb_*' | wc -l), ppt_: $(find jobs_done/ -maxdepth 1 -name 'ppt_*' | wc -l), doc_: $(find jobs_done/ -maxdepth 1 -name 'doc_*' | wc -l)"echo " "
echo " <<<<<<<<<< Jobs_analyze_media ($(date '+%Y-%m-%d %H:%M:%S')) >>>>>>>>>>"; echo "video_: $(find jobs_analyze_media/ -maxdepth 1 -name 'video_*' | wc -l), picture_: $(find jobs_analyze_media/ -maxdepth 1 -name 'picture_*' | wc -l), pdf_: $(find jobs_analyze_media/ -maxdepth 1 -name 'pdf_*' | wc -l), txt_: $(find jobs_analyze_media/ -maxdepth 1 -name 'txt_*' | wc -l), avb_: $(find jobs_analyze_media/ -maxdepth 1 -name 'avb_*' | wc -l), ppt_: $(find jobs_analyze_media/ -maxdepth 1 -name 'ppt_*' | wc -l), doc_: $(find jobs_analyze_media/ -maxdepth 1 -name 'doc_*' | wc -l)"sleep 600 # Wait for 10 minutes (600 seconds)
done - Place yourself inside the jobs_done or Jobs_analyze_media folders and run this command for a break down of the numbers of each type of files finished or in the queue.
echo "video_: $(ls -1 video_* | wc -l), picture_: $(ls -1 picture_* | wc -l), pdf_: $(ls -1 pdf_* | wc -l), txt_: $(ls -1 txt_* | wc -l), avb_: $(ls -1 avb_* | wc -l), ppt_: $(ls -1 ppt_* | wc -l), doc_: $(ls -1 doc_* | wc -l)"
- You can also tail -f each log of interest. Most helpful in the indexing process is the fswatch.log and the processingdriver.log
- Next check the processingdriver.log - look for tickets in progress and active processes, av queue, and doc queue
2024-05-30 05:42:04 INFO tickets_in_progress: 1
2024-05-30 05:42:04 INFO Active processes: ['get_video_embeddings', 'send_audio']
2024-05-30 05:42:03 INFO doc_queue: 7
5. Try checking the database files for anything that happened on a certain date - /opt/caraone_data/log$ cat */database.log | grep "2024-08-19"
6. At the file system level - run a "ls jobs_*" Look for analyze media for existing files still pending processing, and jobs_done for finished jobs
root@AInode-4:/opt/caraone_user1# ls jobs_*
jobs_analyze_media:
pdf_4durzi4ulq6fgiuwyeqt0lr5n picture_27vo4tgcksafs picture_32quvnftz9wuy video_1n2yx7p3lc00 video_3qg3hucqajdvk
pdf_4egnt03okh81irrk6ozo8e47w picture_288h7llexq5aj picture_32wuh7fkemx6y video_1qwr1vmr7zdhe video_3qntwfuvh6rr4
pdf_56sv1acc4u3t2iwbmcm3ujyw7 picture_28shu1cdjqjjh picture_34p80i0vhe48h video_1rgxcuq4zumf4 video_3sn0carpa6rk0
pdf_7nzrvsu76n6bwl8533raiy3sc picture_2c38n785m8ktj picture_37gr7d62m5lxc video_2612ovzusfrb4 video_3stg7pjwjflz4
pdf_833jkm6noa8j575wtwgvzad25 picture_2cmzet8vt630t picture_3axbx0gytfkmg video_261l81z8i7yww video_3stu61x2glji8
pdf_9iva8le3qu2mko1q0wazgwbhp picture_2f8u4uxh1uu4p picture_3crqulzi3ujt5 video_26wskvteyucg video_3vbsali0btf40
pdf_adto6o1s5mq5i3hidsfdzax30 picture_2mi25una1m32m picture_3d9vh442kmvsd video_3cwthd9djfbpc video_3vgog4ykt2k8w
pdf_bqfnv9y1k3dby0x9l0hzfk29r picture_2pac6dav9o87o picture_3t4p5syn49lnm video_3e48l8ajk9awx video_3vm5h314scu0w
pdf_zqs7jz391o4ywz4yhaygf45b picture_2qpgyjnk5msok picture_3uxxryluvzsxj video_3f1j032n81vjs video_3w59b3mujruuw
picture_1z3zipy2vudad picture_2t4augimglb1v txt_21s3cjpmpa1hh7ny1hd1ge92 video_3f3yy7otw5ldu video_3w5dx56mmhakg
picture_21f0ek7kegkf9 picture_2zksfbsokiptv txt_5uuf8l6iu0szvya4gehdttyu7 video_3f3z6rwxc9wsf video_9ptdpxflmagw
picture_21otqlf7a3ip3 picture_2zq472fwc1tqi txt_6p6sfa7oxs1grb8cbd8m0at0x video_3f41d1d4i5ou8 video_h4qe0d1gwadc
picture_228s51yrkrcud picture_2zrkherg372g6 txt_a0xzuah0rz0keudthp2r5ihhu video_3f4pz0niklfr4 video_l6tpibqbk
picture_23pteaiugrd4m picture_303833z1pet55 txt_ez94na7acqqmkjcrg5bhh8rfs video_3k213xraahq0w video_wtz31nv7fh80
picture_24gp2pdlo31ad picture_305wd4r5bcnv4 video_160pbzf8xmtc video_3ne3nd4g2i83k video_wuvll0o722kg
picture_27krxw7ytxnye picture_308j42cbb7mp1 video_1dtg3h3lg5ons video_3nfcoxshqxscr video_yjjkz3o3rtog
jobs_delete:
jobs_done:
pdf_2pt6pk85wzwtyok6zmhxjrmee pdf_a3xg10w0e4nblmxoo0hacsww5 video_13c7mcds41kk video_3vo2frwi4mrc0 video_3w5b9jp6gaeim video_4ng2ta65e8g0
pdf_48vip1rf9kl445tneu8rozcsy pdf_af6vir5tsae5uohk9ei5qe2w5 video_1cdcqpvgmpygw video_3vxgkn5ugcyhd video_3w5b9kcd6o1ks video_97ay3mxz2kg
pdf_53i0gbh8sz0iuby9r67lctq10 pdf_amvmk6beihz9xv8d34t7zz0cu video_1xo6nxjjlbhsw video_3w0dw2m26tlh8 video_3w5b9n7gje7e8 video_fb59c450alfk
pdf_59b8g66loshmystadaimyvx0j pdf_av8amv91vn0gwgr9q411l4d0w video_1xss6j2fovhmo video_3w4rrosn4e2gw video_3w5b9ydynf9xa video_fmxu8ctzymzo
pdf_5sbihzauhib2893kar9f7y93m pdf_awv78z1peug88utk0keba7q4m video_2a6o31mobz8qq video_3w5b998bo8kxr video_3w5bbau648m4c video_h7fl60yavm8
pdf_6zykfiorteytgucyx1e4iwu4o pdf_cxwqb2drxj4pc106cp0wtu143 video_2x3yrdxx04z6o video_3w5b99c9qgc8v video_3w5bizxy03dc0 video_jovk53hwln9c
pdf_7b5gao0u1t1ng75bofbngsv68 pdf_mi3k3f16x9648vsca37b0ps3 video_3ejbp5e6wbpy7 video_3w5b99e8rkqv0 video_3w5byl4gg8ovw video_q9xlyth59hir
pdf_7d2o2y30e3tlgq7u8nvwpnrqg pdf_x251bkch0sa1ceu0031ex0ga video_3k29hzb1ax6v4 video_3w5b99m4us072 video_3w5du63h4jj7k video_s14gvbrqroqo
pdf_7sn64w8mmdkjum9dwcfskync0 pdf_y9e7ape01mpd8nfjiczr5blm video_3ndz52y1p48am video_3w5b99q2vr6da video_3w5e0os2rs2nz video_ve5id72s0hs
pdf_8769kqbheygfzxypmatpra1k1 video_112x5v9ku0236 video_3pzqu3d0c42ki video_3w5b9comm987k video_3w5e104hbs0e8
pdf_9nho42vr2sat7nmttyvynicwe video_132p3vr1x1mao video_3v0tfv50tnge8 video_3w5b9jp6bsovy video_4cigvvwyafk0
jobs_error:
jobs_proxy:
root@AInode-4:/opt/caraone_user1#
Jobs proxy breakdown and jobs done and jobs analyse:
clear; while systemctl is-active --quiet runner; do echo "---------------------------"; echo; echo;
echo "<<<<<<<<<< jobs_analyze_media ($(date '+%Y-%m-%d %H:%M:%S')) >>>>>>>>>>"; echo "video: $(find jobs_analyze_media/ -maxdepth 1 -type f -name 'video_*' | wc -l)"; echo "picture: $(find jobs_analyze_media/ -maxdepth 1 -type f -name 'picture_*' | wc -l)"; echo "pdf: $(find jobs_analyze_media/ -maxdepth 1 -type f -name 'pdf_*' | wc -l)"; echo "txt: $(find jobs_analyze_media/ -maxdepth 1 -type f -name 'txt_*' | wc -l)"; echo "avb: $(find jobs_analyze_media/ -maxdepth 1 -type f -name 'avb_*' | wc -l)"; echo "ppt: $(find jobs_analyze_media/ -maxdepth 1 -type f -name 'ppt_*' | wc -l)"; echo "doc: $(find jobs_analyze_media/ -maxdepth 1 -type f -name 'doc_*' | wc -l)"; echo; echo;
echo "<<<<<<<<<< jobs_done ($(date '+%Y-%m-%d %H:%M:%S')) >>>>>>>>>>"; echo "video: $(find jobs_done/ -maxdepth 1 -type f -name 'video_*' | wc -l)"; echo "picture: $(find jobs_done/ -maxdepth 1 -type f -name 'picture_*' | wc -l)"; echo "pdf: $(find jobs_done/ -maxdepth 1 -type f -name 'pdf_*' | wc -l)"; echo "txt: $(find jobs_done/ -maxdepth 1 -type f -name 'txt_*' | wc -l)"; echo "avb: $(find jobs_done/ -maxdepth 1 -type f -name 'avb_*' | wc -l)"; echo "ppt: $(find jobs_done/ -maxdepth 1 -type f -name 'ppt_*' | wc -l)"; echo "doc: $(find jobs_done/ -maxdepth 1 -type f -name 'doc_*' | wc -l)"; echo; echo;
echo "<<<<<<<<<< jobs_proxy ($(date '+%Y-%m-%d %H:%M:%S')) >>>>>>>>>>"; echo "Tickets: $(find jobs_proxy/ -maxdepth 1 -type f | wc -l)"; echo; sleep 600; done