The Data Mining Forum 
IMPORTANT: This is the old Data Mining forum.
I keep it online so that you can read the old messages.
Please post your new messages in the new forum: https://forum2.philippe-fournier-viger.com/index.php
Can I batch process multiple text files?
Posted by:
tassieTom
Date: October 18, 2019 01:11PM
Good morning,
I am experimenting with skipgrams and TKS. The results are very impressive in text files that I am using. I am wondering if it is possible to batch process a whole folder of txt files instead of entering the input files one at at time in SPMF. Many thanks, Tom
I am experimenting with skipgrams and TKS. The results are very impressive in text files that I am using. I am wondering if it is possible to batch process a whole folder of txt files instead of entering the input files one at at time in SPMF. Many thanks, Tom
Re: Can I batch process multiple text files?
Posted by:
webmasterphilfv
Date: October 23, 2019 06:31AM
Hi,
I am glad you are getting good results!
Currently this feature is not offered. But since you can call TKS from the command line, you could just write a small script to call SPMF for every file of your folder. If you are using Windows, perhaps that a BAT script could do this. Or if you are on Linux I don't know ;-) Or another could be to write a small program to do a loop on all folder.
It is a good feature, and I will think about how I could add this to a future version of SPMF. Thanks for the suggestion.
Best regards,
Philippe
I am glad you are getting good results!
Currently this feature is not offered. But since you can call TKS from the command line, you could just write a small script to call SPMF for every file of your folder. If you are using Windows, perhaps that a BAT script could do this. Or if you are on Linux I don't know ;-) Or another could be to write a small program to do a loop on all folder.
It is a good feature, and I will think about how I could add this to a future version of SPMF. Thanks for the suggestion.
Best regards,
Philippe
Re: Can I batch process multiple text files?
Posted by:
webmasterphilfv
Date: October 23, 2019 06:31AM
Hi,
I am glad you are getting good results!
Currently this feature is not offered. But since you can call TKS from the command line, you could just write a small script to call SPMF for every file of your folder. If you are using Windows, perhaps that a BAT script could do this. Or if you are on Linux I don't know ;-) Or another could be to write a small program to do a loop on all folder.
It is a good feature, and I will think about how I could add this to a future version of SPMF. Thanks for the suggestion.
Best regards,
Philippe
I am glad you are getting good results!
Currently this feature is not offered. But since you can call TKS from the command line, you could just write a small script to call SPMF for every file of your folder. If you are using Windows, perhaps that a BAT script could do this. Or if you are on Linux I don't know ;-) Or another could be to write a small program to do a loop on all folder.
It is a good feature, and I will think about how I could add this to a future version of SPMF. Thanks for the suggestion.
Best regards,
Philippe
Re: Can I batch process multiple text files?
Posted by:
tassieTom
Date: October 23, 2019 08:21PM
Thanks for your response, Philippe. You gave me a clue by mentioning a Bat file for windows. I did a bit of research and managed to get one running which will process all the files in a folder. It works like a charm! Thanks again.
Re: Can I batch process multiple text files?
Posted by:
webmasterphilfv
Date: October 24, 2019 08:11AM
Hi
I am glad to know that you have done it successfully with a BAT file!
Would you mind posting the BAT script? Maybe it could be useful to others!
Best regards,
Philippe
I am glad to know that you have done it successfully with a BAT file!
Would you mind posting the BAT script? Maybe it could be useful to others!
Best regards,
Philippe
Re: Can I batch process multiple text files?
Posted by:
tassieTom
Date: October 24, 2019 01:13PM
Hi Philippe,
Here is the basic version of a BAT file that will process multiple .text files with TKS in a folder. For simplicity there is no directory checking, so SMPF and the .text files need to be in the same folder.
for %%f in (*.text) do (
echo %%~nf
java -jar spmf.jar run TKS "%%~nf.text" "%%~nfOutput.txt" 50 1 4 "" 1 0
)
That's it!
Hope it's useful to someone, it has saved my day.
By the way, here is a way to rename all your .txt files in a folder to .text with a BAT file too, this script could be added to the beginning of the above script.
ren *.txt *.text
Regards,
Tom
Here is the basic version of a BAT file that will process multiple .text files with TKS in a folder. For simplicity there is no directory checking, so SMPF and the .text files need to be in the same folder.
for %%f in (*.text) do (
echo %%~nf
java -jar spmf.jar run TKS "%%~nf.text" "%%~nfOutput.txt" 50 1 4 "" 1 0
)
That's it!
Hope it's useful to someone, it has saved my day.
By the way, here is a way to rename all your .txt files in a folder to .text with a BAT file too, this script could be added to the beginning of the above script.
ren *.txt *.text
Regards,
Tom
Re: Can I batch process multiple text files?
Posted by:
webmasterphilfv
Date: October 25, 2019 08:09PM
Hi Tom,
That is great! Thanks for sharing it!
Best,
Philippe
That is great! Thanks for sharing it!
Best,
Philippe