sa_userconf_maps and sa_username_maps syntax question

[Matt Goodman]

Hello all,



I am interested in two new features that got added to amavisd-new in version
2.7.0



Taken from the Release Notes (2.7.0)



- per-recipient (or per- policy bank) SpamAssassin configuration files or

SQL configuration sets are supported (@sa_userconf_maps), and
per-recipient

SQL Bayes database usernames (@sa_username_maps);



However, I don't see are any examples of how this configuration is enabled,
or more specifically what syntax the configuration options are looking for.
I gather that they are "maps" but I really don't understand how maps are
configured. Presently I have the @lookup_sql_dsn configured to check the
database which Postfixadmin is adding users to. I'd like to have my users
store per-user bayes information and per-user spamassasin information inside
this database also.



Did these options merely get added and will become documented/utilized at a
later point? Or is all the information already there, and I'm just not
seeing it?



Update: I did find these two config options in the amavisd.conf-default.bz2
doc file



# @sa_userconf_maps = ();

# @sa_username_maps = ();



Alas, they are empty and I am not sure what to enter there to "tell"
amavisd-new to look up in the SQL database.



I appreciate any replies.



Matt

[Matt Goodman]

Thank you very much for your reply. I understand a little bit better now. However, in example one below you listed a single domain name using the method ‘sql’. What if I have dozens of domains, with them being added/removed somewhat regularly by Postfix Admin (I can perceive adding at least 1-2 domains per month) to this system. Isn’t there a way to tell amavisd-new that all domains in the lookup_sql_dsn? The table is actually called ‘domain’ inside the ‘postfixadmin’ db.



For @sa_username_maps – I do not understand the syntax of that at all. Could you explain it in just a little more detail so I actually understand what’s being done, as opposed to just copying your example. Many thanks.



Thank you for the link to the patch, that will be helpful. And yes I have followed the SpamAssassin SQL README and have my db schema as-per those documents. A table called ‘userpref’ with one line per SA preference. I have even tested this using spamc and validated that the SQL configuration is good.



I am just a little lost as to the @sa_username_maps syntax (looks like regex to me), and I will read through the patch to try and determine what amavisd-new is looking for. In the meantime, if someone can help me understand how one might use all the domains in the postfixadmin table instead of one domain per line of @sa_userconf_maps, that would be helpful.



Matt



Sent: Thursday, September 01, 2011 11:21 AM
To: Matt Goodman
Subject: Re: sa_userconf_maps and sa_username_maps syntax question



Hello Matt,

For sql user_prefs ( per user config )::

I had to add

1. @sa_userconf_maps = (
{
'.domain-t1.ro' => 'sql:',
}
);

@sa_username_maps = new_RE (
[ qr'^([^@]+@.*)'i => '${1}' ]
);

2. this patch

http://lists.amavis.org/pipermail/amavis-users/attachments/20110812/756a66cf/attachment.obj

You have to read about SPAMMASSASSIN SQL userprf to know howto setup database and userpref table.

[Renato Botelho]

Hello Matt

Post by Matt Goodman
Thank you very much for your reply. I understand a little bit better now.
However, in example one below you listed a single domain name using the
method ‘sql’. What if I have dozens of domains, with them being
added/removed somewhat regularly by Postfix Admin (I can perceive adding at
least 1-2 domains per month) to this system. Isn’t there a way to tell
amavisd-new that all domains in the lookup_sql_dsn? The table is actually
called ‘domain’ inside the ‘postfixadmin’ db.

If all domains are using sql, you can have:

@sa_userconf_maps = (
   {
      '.*' => 'sql:',
   }
);

Post by Matt Goodman
you explain it in just a little more detail so I actually understand what’s
being done, as opposed to just copying your example. Many thanks.

In this case, since it is a single domain, i have personal userprefs
set just for uset, without domain, i mean, if email is ***@foo.com
prefs on SQL is set just for user, so, this map says exactly it, for
***@foo.com, look just for user at SQL database.

@sa_username_maps = new_RE (
  [ qr'^([^@]+@.*)'i => '${1}' ]
);

Post by Matt Goodman
Thank you for the link to the patch, that will be helpful. And yes I have
followed the SpamAssassin SQL README and have my db schema as-per those
documents. A table called ‘userpref’ with one line per SA preference. I have
even tested this using spamc and validated that the SQL configuration is
good.

What that patch does is making amavis respect personal required_score
setting instead of use a global value for everyone, and the version sent
to this thread has a small issue if you use it with spamassassin < 3.0.

Post by Matt Goodman
to me), and I will read through the patch to try and determine what
amavisd-new is looking for. In the meantime, if someone can help me
understand how one might use all the domains in the postfixadmin table

[Matt Goodman]

OK. So is it correct in saying the following:

@sa_userconf_maps matches SA user preferences by domain
@sa_username_maps matches SA Bayes database by email address

As I understand it - in your regex example, amavisd-new will simply match
the recipient email address to the corresponding record in the bayes
database, correct? If this is not correct, any insight would be appreciated.
Again, I am not looking for a shortcut - only to understand the way the
subroutines were intended for use.

Ultimately I am trying to have a flexible virtual user email system with
per-user spamassassin preferences and per-user Bayesian filters, allowing
people to "train" their own Bayes tokens by "learning" messages which they
mark as Spam and "learning" those which are ham, while storing all that
information into a SQL database. The assumption here is that the Bayesian
classification will be more accurate to the individual user, rather than a
one-size-fits all for Bayes seen/tokens.

However, amavisd-new is my chosen mail filter system, simply because of how
flexible it is, and so far the help I have received here indicates that I
have made the right choice. I will share any findings that I learn back to
the community. Thank you again.

Matt

-----Original Message-----
From: Renato Botelho [mailto:***@gmail.com]
Sent: Friday, September 02, 2011 12:07 AM
To: Matt Goodman
Cc: amavis-***@amavis.org
Subject: Re: sa_userconf_maps and sa_username_maps syntax question

Hello Matt

Post by Matt Goodman
Thank you very much for your reply. I understand a little bit better now.
However, in example one below you listed a single domain name using
the method ‘sql’. What if I have dozens of domains, with them being
added/removed somewhat regularly by Postfix Admin (I can perceive
adding at least 1-2 domains per month) to this system. Isn’t there a
way to tell amavisd-new that all domains in the lookup_sql_dsn? The
table is actually called ‘domain’ inside the ‘postfixadmin’ db.

If all domains are using sql, you can have:

@sa_userconf_maps = (
   {
      '.*' => 'sql:',
   }
);

Post by Matt Goodman
Could you explain it in just a little more detail so I actually
understand what’s being done, as opposed to just copying your example.

Many thanks.

In this case, since it is a single domain, i have personal userprefs set
just for uset, without domain, i mean, if email is ***@foo.com prefs on SQL
is set just for user, so, this map says exactly it, for ***@foo.com, look
just for user at SQL database.


@sa_username_maps = new_RE (
  [ qr'^([^@]+@.*)'i => '${1}' ]
);

Post by Matt Goodman
Thank you for the link to the patch, that will be helpful. And yes I
have followed the SpamAssassin SQL README and have my db schema as-per
those documents. A table called ‘userpref’ with one line per SA
preference. I have even tested this using spamc and validated that the
SQL configuration is good.

What that patch does is making amavis respect personal required_score
setting instead of use a global value for everyone, and the version sent to
this thread has a small issue if you use it with spamassassin < 3.0.

Post by Matt Goodman
regex to me), and I will read through the patch to try and determine
what amavisd-new is looking for. In the meantime, if someone can help
me understand how one might use all the domains in the postfixadmin

helpful.

--
Renato Botelho

[Matt Goodman]

Thank you for the insight. I examined the patch file that was presented and
I have to go a different direction. Since required_score is somewhat handled
already via the policy system (although users cannot specify their own
'score' they can indeed change policies, which in turn affects score), I
think I'm going to focus on per-user Bayes.

What part of amavisd-new goes and checks per-user bayes? I see nothing in
earlier examples nor in the patch which address having amavisd-new handle
per-user Bayes classification. Or is that solely done through SA's
configuration? If no changes are required in amavisd-new to get per-user
Bayes working, then I apologize. I can get SA's /etc/spamassassin/local.cf
to use the per-user Bayes. I'm just new to amavis and assume that amavis has
to handle checking against the per-user Bayes database. Is this correct?

So I will abandon sa_userconf_maps and sa_username_maps and go straight to
per-user Bayes. That should be enough, when combined with the flexibility of
the amavisd-new policies via SQL to accomplish what I need. I just have to
figure out whether anything is required on the amavis side of things.

Thanks,

Matt

-----Original Message-----
From: Renato Botelho [mailto:***@gmail.com]
Sent: Friday, September 02, 2011 12:07 AM
To: Matt Goodman
Cc: amavis-***@amavis.org
Subject: Re: sa_userconf_maps and sa_username_maps syntax question

Hello Matt

Post by Matt Goodman
Thank you very much for your reply. I understand a little bit better now.
However, in example one below you listed a single domain name using
the method ‘sql’. What if I have dozens of domains, with them being
added/removed somewhat regularly by Postfix Admin (I can perceive
adding at least 1-2 domains per month) to this system. Isn’t there a
way to tell amavisd-new that all domains in the lookup_sql_dsn? The
table is actually called ‘domain’ inside the ‘postfixadmin’ db.

If all domains are using sql, you can have:

@sa_userconf_maps = (
   {
      '.*' => 'sql:',
   }
);

Post by Matt Goodman
Could you explain it in just a little more detail so I actually
understand what’s being done, as opposed to just copying your example.

Many thanks.

In this case, since it is a single domain, i have personal userprefs set
just for uset, without domain, i mean, if email is ***@foo.com prefs on SQL
is set just for user, so, this map says exactly it, for ***@foo.com, look
just for user at SQL database.


@sa_username_maps = new_RE (
  [ qr'^([^@]+@.*)'i => '${1}' ]
);

Post by Matt Goodman
Thank you for the link to the patch, that will be helpful. And yes I
have followed the SpamAssassin SQL README and have my db schema as-per
those documents. A table called ‘userpref’ with one line per SA
preference. I have even tested this using spamc and validated that the
SQL configuration is good.

What that patch does is making amavis respect personal required_score
setting instead of use a global value for everyone, and the version sent to
this thread has a small issue if you use it with spamassassin < 3.0.

Post by Matt Goodman
regex to me), and I will read through the patch to try and determine
what amavisd-new is looking for. In the meantime, if someone can help
me understand how one might use all the domains in the postfixadmin

helpful.

--
Renato Botelho

[Mark.Martinec+ (Mark Martinec)]

Matt,

Post by Matt Goodman
I examined the patch file that was presented and
I have to go a different direction. Since required_score is somewhat
handled already via the policy system (although users cannot specify their
own 'score' they can indeed change policies, which in turn affects score),
I think I'm going to focus on per-user Bayes.

Makes sense, per-user Bayes is much more efficient than per-user
SpamAssassin preferences/config.

Post by Matt Goodman
What part of amavisd-new goes and checks per-user bayes? I see nothing in
earlier examples nor in the patch which address having amavisd-new handle
per-user Bayes classification.

There is a 80+ line description of @sa_username_maps and @sa_userconf_maps
with examples in the 2.7.0 release notes.

Search for sections starting with:
- per-recipient (or per- policy bank, or global) SpamAssassin configuration
and
- per-recipient (or per- policy bank) SpamAssassin SQL database usernames

Post by Matt Goodman
Or is that solely done through SA's configuration?

Before a call to SpamAssassin, amavisd tells SpamAssassin which username
to use. The rest (SQL lookup etc.) is done by SpamAssassin.

Post by Matt Goodman
If no changes are required in amavisd-new to get per-user
Bayes working, then I apologize.

No changes should be necessary, apart from configuring @sa_username_maps
in amavisd.conf.

Post by Matt Goodman
I can get SA's /etc/spamassassin/local.cf
to use the per-user Bayes. I'm just new to amavis and assume that amavis
has to handle checking against the per-user Bayes database. Is this
correct?

When per-recipient bayes is needed, amavisd calls SpamAssassin possibly
multiple times for multiple recipients of a message, once for each unique
username. Before the call the username is passed to SpamAssassin,
the rest is SpamAssassin's task.

Mark

[Matt Goodman]

Ah thanks, I am reviewing the release-notes now.

However, what I am reading is that the example listed for @sa_username_maps
statically maps usernames which is a smaller scale than I am trying to
implement. With a larger scale system, how do I tell @sa_username_maps to
pass the email address supplied from the RCPT TO as the SA username?

Or is that what the earlier reference to the following will do (provided by
Florin Portase on 09/01/2011):

@sa_username_maps = new_RE (
[ qr'^([^@]+@.*)'i => '${1}' ]
);

And finally, will I need to have @sa_userconf_maps set to "sql:" if I only
wish to do per-user bayes? Would this potentially cause any other problems
if we enable this setting but do not use any per-user preferences?

For reference:

All my users will have their own custom bayes without exception. We do not
wish to use per-domain or per-policy-bank bayes, per-user only. My SA/bayes
SQL configuration is sound and currently working - however, only one bayes
database is in the table, and it is owned by user 'amavis'.

-----Original Message-----
From: amavis-users-bounces+matt=***@amavis.org
[mailto:amavis-users-bounces+matt=***@amavis.org] On Behalf Of Mark
Martinec
Sent: Thursday, September 08, 2011 7:50 AM
To: amavis-***@amavis.org
Subject: Re: sa_userconf_maps and sa_username_maps syntax question

Matt,

I examined the patch file that was presented and I have to go a
different direction. Since required_score is somewhat handled already
via the policy system (although users cannot specify their own 'score'
they can indeed change policies, which in turn affects score), I think
I'm going to focus on per-user Bayes.

Makes sense, per-user Bayes is much more efficient than per-user
SpamAssassin preferences/config.

What part of amavisd-new goes and checks per-user bayes? I see nothing
in earlier examples nor in the patch which address having amavisd-new
handle per-user Bayes classification.

There is a 80+ line description of @sa_username_maps and @sa_userconf_maps
with examples in the 2.7.0 release notes.

Search for sections starting with:
- per-recipient (or per- policy bank, or global) SpamAssassin
configuration and
- per-recipient (or per- policy bank) SpamAssassin SQL database usernames

Or is that solely done through SA's configuration?

Before a call to SpamAssassin, amavisd tells SpamAssassin which username to
use. The rest (SQL lookup etc.) is done by SpamAssassin.

If no changes are required in amavisd-new to get per-user Bayes
working, then I apologize.

No changes should be necessary, apart from configuring @sa_username_maps in
amavisd.conf.

I can get SA's /etc/spamassassin/local.cf to use the per-user Bayes.
I'm just new to amavis and assume that amavis has to handle checking
against the per-user Bayes database. Is this correct?

When per-recipient bayes is needed, amavisd calls SpamAssassin possibly
multiple times for multiple recipients of a message, once for each unique
username. Before the call the username is passed to SpamAssassin, the rest
is SpamAssassin's task.

Mark

[Renato Botelho]

On Thu, Sep 8, 2011 at 2:50 PM, Mark Martinec

Post by Mark.Martinec+ (Mark Martinec)
Matt,

Post by Matt Goodman
I examined the patch file that was presented and
I have to go a different direction. Since required_score is somewhat
handled already via the policy system (although users cannot specify their
own 'score' they can indeed change policies, which in turn affects score),
I think I'm going to focus on per-user Bayes.

Makes sense, per-user Bayes is much more efficient than per-user
SpamAssassin preferences/config.

Hello Mark,

I understand that it's more efficient, but, in some scenarios you *need* to
need user the powers to select their own required_score. That was the reason
i wrote that patch.

Do you see any real problem on the changed i made? I mean, do you think
it could break something?

And since amavisd-new is a one-file program, and this is a really big file, do
you consider applying this patch if it don't break anything? I'm asking because
maintain a custom patch for a so big file is a hard task sometimes.

Regards

[Mark.Martinec+ (Mark Martinec)]

Renato,

Post by Renato Botelho
I understand that it's more efficient, but, in some scenarios you *need* to
need user the powers to select their own required_score. That was the
reason i wrote that patch.

Makes sense, although I'd probably want to reverse the logic:
if a lookup on @spam_tag2_level_maps returns undef, then fall back
to a default 'required_hits' as provided by a SpamAssassin's config.

This way it will remain backwards compatible with existing configurations.

Post by Renato Botelho
Do you see any real problem on the changed i made? I mean, do you think
it could break something?

Probably not.

Post by Renato Botelho
And since amavisd-new is a one-file program, and this is a really big file,
do you consider applying this patch if it don't break anything? I'm asking
because maintain a custom patch for a so big file is a hard task
sometimes.

Sure, after reversing the logic.

Mark

[Matt Goodman]

For the @sa_username_maps I have the following provided by Florin:

@sa_username_maps = new_RE (
[ qr'^([^@]+@.*)'i => '${1}' ]
);

As he pointed out, he uses only a single domain on his mail server and this
expression maps the 'user' portion ***@domain.com and sends that as the SA
username. How would I modify this so that I pass '***@domain.com' as the SA
user? I've got the SQL portion up and running, I just don't have the
username maps set up properly. I've been trying for a few days to figure out
how to do this but I am unsure :(

Any insight would be appreciated.

Matt

-----Original Message-----
From: amavis-users-bounces+matt=***@amavis.org
[mailto:amavis-users-bounces+matt=***@amavis.org] On Behalf Of Mark
Martinec
Sent: Friday, September 09, 2011 2:36 PM
To: amavis-***@amavis.org
Subject: Re: sa_userconf_maps and sa_username_maps syntax question

Renato,

Post by Renato Botelho
I understand that it's more efficient, but, in some scenarios you
*need* to need user the powers to select their own required_score.
That was the reason i wrote that patch.

Makes sense, although I'd probably want to reverse the logic:
if a lookup on @spam_tag2_level_maps returns undef, then fall back to a
default 'required_hits' as provided by a SpamAssassin's config.

This way it will remain backwards compatible with existing configurations.

Post by Renato Botelho
Do you see any real problem on the changed i made? I mean, do you
think it could break something?

Probably not.

Post by Renato Botelho
And since amavisd-new is a one-file program, and this is a really big
file, do you consider applying this patch if it don't break anything?
I'm asking because maintain a custom patch for a so big file is a hard
task sometimes.

Sure, after reversing the logic.

Mark