SuckMT v.0.57

Version 0.57 was released on 2004-10-01

(C) 1999-2003 by Niels Basjes


Frequently Asked Questions
Download SuckMT
Whats new in version 0.57
The TODO list for future versions.


What ?

SuckMT is intended to be a multithreaded replacement for suck.
Note that SuckMT was inpired by suck but SuckMT has absolutely NO CODE IN COMMON with suck. !!

Why ?

The main problem I personally have with the original suck (currently maintained by Robert A. Yetman) is that it is single threaded.

At first glance this sounds like a silly reason to start a new implementation from scratch but I think it isn't. At home I have a 56K (== 5KB/s real download speed) telephone connection with a charge for each connected second (as do most people in the Netherlands). The problem I ran into is that when downloading news messages the original suck doesn't go above about 2KB/s. This means that I needed roughly 2.5 times more online time than was stricktly required.

When asking in nl.comp.os.linux on how to solve this problem people suggested to basically create two separate installations of suck and to manually spread the newsgroups between the two. This meant to me that something like this should be included into suck. After having had a quick look at the suck source code I descided that doing it from scratch would be better for my health. ;-)

How ?

The basic idea of SuckMT is that it has several Command handlers each with its own NNTP connection (all to the same server). All commandhandlers share the same threadsafe queue which contains a "todo-list". When a commandhandler is idle it gets the next command from the queue and starts to execute it. This way SuckMT automatically balances the download of all NEWS items over the available NNTP socket connections thus actually using all the available bandwidth of my 5KB/s modem line. When properly configured the actual maximum lies at about 8KB/s because a modem can do some pretty good compression on plain ASCII information like usenet messages.

Where ?

You can download SuckMT right here! and also some sample scripts.
SuckMT is released under the GNU Public License.

If you're using SuckMT please send me an email with your comments, likes and dislikes.

Don't forget to check the FAQ

Download SuckMT

Version Release Date Source Precompiled Binaries
All platforms Linux-i386 Windows NT 4-i386
Dynamically linked Statically linked Statically linked
0.57 2004-10-01 .tar.gz   .src.rpm .rpm  
0.56 2004-07-28 .tar.gz   .src.rpm .rpm  
0.55 2003-04-30 .tar.gz   .src.rpm .rpm   .tar.gz .tar.gz
0.54 2001-08-28 .tar.gz   .src.rpm .rpm   .tar.gz .tar.gz .zip
0.53 2000-10-23 .tar.gz   .src.rpm .rpm .tar.gz .zip
before 0.53 1999-2000 All files including the old ones

Sample Scripts

Because some people had problems posting to their local INN servers I decided to post my own config files and scripts related to this issue on this website. These files are my actual config files and they really work for me but they may not work for you mainly because you probably have a lot of things in different places. I have an older version of suck installed (for rpost) and these scripts assume it to be installed. These scripts have now been included in the 0.55 distribution but can also be downloaded here as SuckMT-sample-scripts.tar.gz.


History

There is also a TODO list of items I want to put in future versions of SuckMT.

Changes from 0.56 to 0.57
- Fixed: Serious bug in the INI file read code. Thanks to Bob Jaques for pointing it out to me. This bug effectively made suckmt unusable on many platforms. The strange thing is that the effected code has been in there since almost the very first version. I should have spotted this when making 0.56 but back then I only focussed on the makefiles and related scripts.

Changes from 0.55 to 0.56
In general this version has not changed the functionality at all. This is just a minor fixed version that allows for succesfull compilation on newer platforms (like Fedora Core 2).
- Fixed: Two issues regarding changed behaviour of automake:
- Added AC_PROG_CC to configure.in_ even though the C compiler is not used. Compilation would fail without it.
- Changed CFLAGS and CXXFLAGS into AM_CFLAGS and AM_CXXFLAGS dded initial support for BSD and Cygwin
- Fixed: Minor issue in the spec file (for building RPMS).

Changes from 0.54 to 0.55
In general this version has not changed the functionality significantly. Most changes are related to makefiles, RPM building, changed behaviour of the GCC compiler and packaging.
- Fixed: Some minor compiler warnings
- Updated: Added initial support for BSD and Cygwin
- Updated: Handling of the 423 error state (no such article number)
- Fixed: Some compiler problems with the FreeBSD port (Thanks to Charles F. Randall).
- Updated: Changed makefiles so they automatically force the usage of GNU make
- Fixed: Some errors and warnings that came to light with gcc3 (Yes, these were bugs in the code).
- Fixed: Some minor issues regarding GCC 3.2 as provided with Redhat 8.0 and 9.0
- Improved: It is now possible for any regular user to do make rpm.
- Improved: Included some of the improvements as provided by PLD.
- Added: Sample scripts are now part of the standard distribution

Changes from 0.53 to 0.54
- Added: Compiler flags for IRIX64 and Solaris (not fully tested).
- Added: Option to remove the ^M from the lines in the downloaded messages.
- Changed: Completely different regular expression library. Now uses the regex++ from www.boost.org which was created by Dr John Maddock. It is currently only enabled on Win32 due to an internal compiler error on my system (SuSE 6.1).
- Fixed: Usage of namespaces and C++ include files. I found that I really did it wrong in the previous versions.compiler:

Changes from 0.52 to 0.53
- Fixed: Uninitialized Memory Read in regular expression library.
- Changed: Can now use a "no posting" server (Thanks to Stephane Lentz)
- Updated: Some unimportant code layout stuff.
- Added: Compiler flags for IRIX (not fully tested).
- Added: The ini file can now contain comments that will be written back again.
- Fixed: forgotten #include <unistd.h> in DuplicatesChecker.cpp which didn't show up in SuSE 6.1
- Added: Maximum messages per group download feature if over limit.
- Changed: Default config file (if nothing is specified) is now /etc/suckmt.ini
- Changed: Default value for "Send MODE READER to the server" is now True.
- Improved: Installation prcedure now automagically includes a working config file.
- Fixed ?: Removed alarm call from GenericSocket.cpp, perhaps this solves the SIG_ALRM problem.
- Fixed ?: Removed regfree call from HeaderMatcher.cpp, perhaps this solves some unexplained crashes. It does introduce memory leaks at exit.
- Fixed: The mutexes in MultiStream.cpp which still allowed some race conditions.

Changes from 0.51 to 0.52
- Added: Retry option when the server says "Connection refused". The default value doesn't change 0.51 behaviour.
- Fixed: In case some of the connections fail strange things happened. This exotic situation is handled better now.
- Added: Better handling of "430 No such article"
- Fixed: Some compiler errors in Mandrake 6.0 (querying www.deja.com helps)
- Changed: Suppress the warnings from compiling the regex.c file (It's clear I didn't write that monstrosity).
- Fixed: Some of the statistics didn't add up correctly.
- Fixed: The messages on the screen from the different threads will no longer mix through each other even if one thread uses \r to stay on the same line.
- Fixed: Invalid message of 1 restart ID when none were present (Win32).
- Fixed: Didn't do a closesocket at the end.
- Changed: The NNTPGetArticleCommand can now be restarted when the connection failed.
- Changed: The way the connections got their number for the messages.

Changes from 0.50 to 0.51
- Fixed: The restart file didn't include the killed messages.
- Fixed: Constant size memory leak (unimportant but I fixed it anyway).
- Changed: The makefiles have been improved and it's now possible to enable debugging without having to change the makefiles.

Changes from 0.41 to 0.50
- You can check the progress when you do ps from a shell.
- Fixed Bug reading a value in the ini file that contained an = in the value.
- Added Restart file so it is safe to make the temp dir empty after an aborted download session.
- Added the headername "ArticleBody" can be used to create kill/keep rules that check for the actual article content.
- Added MultiStreams, all messages pass through a single stream type that can be configured to pass the information to the cerr, cout , the current logfile and the syslogd.
- Rewrite: (Almost) All messages use the LogStreams now.
- Changed: The -init option now updates the INI file if it already exists.
- Fixed: It is now possible to create an RPM and SRPM file.

Changes from 0.40 to 0.41
- SERIOUS BUG in 0.40 !!!! All statistics will be reset to 0 and all global kill rules will be set to -1 due to a serious bug in the read routine.

Changes from 0.3j to 0.40
- Regular expression matching of kill/keep rules.
- Case insensitive matching of kill/keep rules.
- Simple scoring system for all kill/keep rules.
- Added "Minimum Lines in a Message" as requested by Petri Laihonen.

Changes from 0.3i to 0.3j
- Changed the command syntax from just \n to \r\n and made this a centralized issue. I hope this fixes some problems that were reported regarding the authinfo.
- Added a simple debug option to track the socket commands.
- Created somewhat better makefiles.

Changes from 0.3h to 0.3i
- Created transparant boolean readout for IniFile.
- Made the NNTProxy query its parameters from an IniFile.
- Added the option for MODE READER
- Added the option for username and password to login using AUTHINFO
- Fixed lockup problem when aborting.

Changes from 0.3g to 0.3h
- Counting the messages missed some messages when errors ocurred

Changes from 0.3f to 0.3g
- Fixed a serious memory leak in NNTPGetArticleCommand when aborting.
- Minor fixes and resolution of compiler warnings (on SuSE 6.1: no warnings at all).
- Fixed some compilation problems that occur on Slackware (thanks to Dennis Bijwaard).

Changes from 0.3d to 0.3f
- Keep headers to overrule the kill headers.
- Kill logfile option.
- Re-port to NT to do some additional testing and debugging.

Changes from 0.3c to 0.3d
- Fixed a bug regarding the statistics number in the suckmt.ini file.
- Optimized the code a little (some kill checks moved to before downloading the header).

Version 0.3c
Does:
- work for me.
- Run on Linux 2.2.5 (SuSE 6.1)
- Download several newsgroups.
- Support transparent restarts.
- Generate output that can be post-processed by the same software as the original suck (I use innxmit).
- Supports killing on all kinds of headers and things like the maximum number of lines in the message and the maximum number of groups a message is crossposted to.
- Is portable to Windows NT (I did this only for testing and ensuring code portability).
- Include a installation guide.

Doesn't:
- Include a manual.
- Include a README file.
- Have tidy makefiles.
- do lots of other things which I don't realize yet....

Build and install ?

Just unpack the archive and run make in the source directory.
You'll get a single executable and that is everything.
Run the executable and you should be able to figure out the rest yourself.
The INI file that is generated can be editted using any plain text editor, if you see the file you'll know what the format is.
Read the INSTALL file for more detailled information.

Frequently Asked Questions

This is the FAQ for SuckMT. If you have a question not handled in the FAQ just mail me using SuckMT@Basjes.nl.

Planned enhancements for future versions of SuckMT.

This is a list of items I plan to put in furture versions of SuckMT.
There are three sections here: "New features", "Improvements in the robustness/quality" and "Research into possible new features/ideas".

Before you get you hopes up of seeing any of these features in the next few weeks: I'm currently juggeling a full time (40 hour/week) job, a parttime (25 hour/week) university study and a new appartment that is in desparate need of decorating. Guess what... SuckMT will have to wait a little while (for now).
I you feel like adding to or changing something to SuckMT don't hesitate to fiddle around with the code. If you have a good idea for a new feature that is not listed below don't hesitate to contact me.

Planned Features Planned improvements regarding the robustness and quality of SuckMT Research into new features

Comments

If you like, hate, have good ideas or have something else to tell me about SuckMT don't hesitate to let me know.



SuckMT is Copyright (C)1999-2001 by Niels Basjes