|
1. Introduction
This paper describes the
implementation of a content-adapting personalization system at the
website of the second largest Swedish daily newspaper, the Svenska
Dagbladet (SvD, 2001). The system utilizes content descriptions provided
by the journalists to personalize information to the readers. The
journalists categorize the material according to a modified news
categorization standard and the readers are exposed to a special
selection that is personalized on an individual basis. The reader’s
behavior is recorded and used to optimize the business objectives.
With an increasing information
supply, the use of personalization promises timesaving and better
matching of a given service or product to preference (Peppers et al.,
1999). The term personalization is used to describe the
computer-supported process of adapting an information flow and its
presentation in real time to each individual user’s behavior (Wallin et
al., 1998).
Businesses recognize an opportunity
to increase user satisfaction by learning from the user’s behavior and
thus being able to pinpoint individual preferences and adapt the
communication accordingly to increase business efficiency. To increase
revenues the personalization system needs to influence user behavior so
as to increase user activity (i.e. frequency) on the website. The
primary source of revenues for the news services industry is
advertising; the larger the audience (traffic volume), the higher the
revenues. With the increasing cost of investment and operation of
information technology systems, the need for cost-effective solutions
increases and personalization systems must maintain cost efficient
operation. (Peppers et al., 1999).
Examples of personalization of news
services on the Internet reach back to early experiments in the
beginning of the 1990s (Chesnais et al., 1995). The NY Times’ Internet
edition has successfully been using methods to adapt information to
different segments (Saunders, 2002). A popular method of adapting
information is called content filtering (Foltz et al., 1992) and refers
to the concept of selecting content for a user based on the user’s
characteristics and a description of the content (Lassila, 1997).
Turpeinen et al. (Turpeinen et al., 1996) describe a system for a
personalized news service based on semantic descriptions of the news
items and corresponding user profiles. Turpeinen concludes that
personalization is important for news producers because it adds value to
their users. The SmartPush project (Jokela et al., 2001) found an
operational method of producing structured semantic metadata for news
services in a manner that worked well with personalization.
2. Objective
The primary objective of this paper
is to evaluate if personalization provides a positive effect on
measurable business objectives. The secondary objective is to find the
parameters and settings that offer the best effects on business.
3. Operational functionality of the
personalization system
3.1 The basics of the
personalization system
Personalization is based on the
behavior of the user. Behavioral data is collected when a user interacts
with different information objects in the system (e.g. an editorial
article). Multiple interactions are gathered and saved as the user’s
individual profile. When the system performs a personalization, the
user’s profile is compared to profiles of the information objects and
they are sorted accordingly. (Wallin, 2000).
3.2 System overview
All notations in brackets () refer
to Figure 1. The personalization system (E) is a stand-alone system that
is connected to the web system of SvD (A).
3.2.1 System implementation at
the SvD (A) consists of:
(B) Presentation layer. At
the SvD this is the web page that the user sees.
(C) Application layer. At the
SvD this is a COM component together with an ASP-script (Winnick Cluts,
1997)
(D) Data layer. At the SvD
this is an SQL database.
(E) Personalization system.
3.2.2 User identification
The personalization system receives
the user identification with every interaction between the
personalization system and the SvD website. The personalization system
uses SvD’s user identification mechanism which consists of an anonymous
unique identifier in the form of a cookie (Kristol et al., 1997) that is
placed on a user’s computer the first time the user visits the SvD
website.
3.2.3 Profiles and metadata
The personalization system uses two
kinds of metadata profiles (Lennstrand et al., 1999) for the
personalization process.
§ Article profile – describes the editorial
articles.
§ User profile – describes the individual user’s
characteristics based on historical behavior.
The descriptions are stored as
vectors (Wallin et al., 2000) representing different aspects of the
article (Wallin et al., 1998).
3.3 Processes
All notations in brackets () refer
to Figure 1.
3.3.1 Personalization of the
website:
(1) A user enters the SvD website and requests a
personalized object (e.g. editorial articles).
(2) The application layer (C) requests the desired
articles from the data layer (D).
(3) The requested articles are returned to the
application layer (C)
(4) The application layer sends the articles to the
personalization system (E).
(5) The articles are sorted (Wallin et al., 2000)
according to the user’s profile.
(6) The personalized (sorted) articles are returned
to the application layer (C).
(7) The application layer (C) delivers the
personalized articles to the presentation layer (B). The user receives a
personalized web page.
Figure 1: System overview
3.3.2 Recording the user’s
behavior:
I. A user enters the SvD
website and requests an object (e.g. some editorial articles) that the
personalization system has knowledge of.
II. The application layer
(C) sends a notification to the personalization system (E) that a
specific user has interacted with a specific article.
III. The personalization
system (E) records the event (Wallin et al., 2000)
3.3.3 Personalization
(sorting)
The process of sorting (5) the
articles according to the users profile is done with an exchangeable
algorithm (Wallin et al., 2000). The algorithm used at SvD is of a
shortest distance type (Russel et al., 1995).
4. Implementation at the SvD
During one week (11/19/2001 to
11/25/2001) a test of the personalization system was performed on the
SvD website. The personalization system had been implemented in August
2001.
4.1 Business objectives
SvD’s primary business objective
for the personalization system was to increase the average number of
page impressions a user demanded per session. Page impressions are
defined as the total number of downloads of web pages from a website (KIA,
2003).
4.2 Personalized functions
The selection of the different
functions and areas was made in cooperation with SvD. One of SvD’s
requests was that one area should be personalized on the first web page
to address the problem that the majority of the visitors left the site
after only viewing the first page.
A second area was at the end of an
individual editorial article. The logic behind this function was the
assumption that when a user had finished reading an article, a
personalized selection of articles would increase the chance that the
user would choose to read more.
4.2.1 The first page
In the upper-right hand corner of
the first web page a small area was used to display a section name as
header in a large font and the heading of the latest editorial article
in that section in a smaller font. The smaller text was a hyperlink to
the selected article. The articles used were limited to the three most
recently published articles from the different sections. (Figure 2).
Figure 2.
4.2.2 Article content pages
At the bottom of all article
content pages a list of five other article headers was displayed as
hyperlinks to the respective articles. (Figure 3).
Figure 3.
4.3 Metadata structure
The personalization operation is
based on two metadata descriptions of the news material, (1) Category
(e.g. sports, entertainment, international news etc.) and (2) Geographic
location (e.g. Stockholm, Sweden, Europe, etc.).
For the category description SvD is
using a TT NITF metadata structure (Stadler et al., 2003). The TT NITF
metadata structure originates from the International Press
Telecommunications Council’s (ITPC) commonly used standard for
categorization of news material called NITF (News Industry Text Format)
(Karben, 2003). SvD uses the top-level of the TT NITF and has made minor
modifications to fit their internal information structure. The
geographic location description was retrieved from SvD’s internal
information structure.
A journalist who has written an
article also encodes it with metadata. Some articles from other sources
with compatible TT NITF metadata encoding are imported to SvD.
5. Method
5.1 Delimitation
The results regarding the benefits
of personalization are delimited to quantitative measures without any
qualitative input from the end-users.
5.2 Data collection
To measure the effect of the
personalization system with regards to the business objectives, a method
called bifurcation (Lennstrand et al., 1999) is used to stochastically
divide the users into two or more different groups and then compare the
outcome between the groups and analyze the results. (Wallin et al.,
2000)
The results regarding the
performance of the personalization system were gathered in an
SQL-database in real-time and presented via a report subsystem similar
to OLAP (On-Line Analytical Processing).
5.3 Data selection
During one week (11/19/2001 to
11/25/2001) a test of the personalization system was performed on the
SvD website (SvD, 2001). The personalization system had been implemented
in August 2001. This period was chosen because it was the only one with
minor operational interruptions. The interruptions originated from
difficulties with the integration of a new system into a complex and
heavily burdened system environment. The user profiles originated from
the beginning of the implementation and were not erased at the start of
the test.
The test was limited to three
groups with active personalization and one group with no personalization
(the reference group). The users in the reference group received
articles selected alphabetically rather than based on individual
personalization. A total of only four groups was chosen to reduce the
analytical complexity of the results. The personalization system
stochastically assigned every single user with one of four different
predefined personalization configurations that the user kept throughout
the test period.
Table 1: Personalization
Configurations used.
Table 1: Traffic volume represents
the distribution of the users to the different configurations.
|
Personalization group |
Traffic volume |
Algorithm @ weight |
End update weight |
|
1_33 |
25% |
AL 2.0 @ 33% |
50% |
|
2_67 |
25% |
AL 2.0 @ 67% |
50% |
|
3_100 |
25% |
AL 2.0 @ 100% |
50% |
|
Reference |
25% |
Null |
50% |
The weights in the algorithm field
represent the weight with which the profile representing the user’s
interactions during the active session is merged with the profile
representing the user’s historical behavior, i.e. the volatility of the
system. The same matching algorithm (AL 2.0) was used in all
configurations.
The End update weight represents
the weight with which the user’s historical profile is updated at the
end of the session to reflect the session behavior, i.e. the articles
the user has interacted with during the session. The End update weight
parameter was not the subject of the test and therefore heuristically
chosen.
The implication of the different
algorithm weights is that a lower number (e.g. 33%) means that the
personalization system takes greater consideration of the historical
behavior. A setting of 100% provides an entirely session-dependent
system, or in other words an extremely volatile system.
During the test, all interactions
performed by the users were logged, including session length and
requests for articles
5.4 Validation
The integrity of the built-in
report system of the personalization system was verified by random
manual comparison calculations. In this test no data has been subtracted
or corrected.
6. Results from SvD
Sessions = Total number of times
web pages have been downloaded from a website by a web browser on a
coherent occasion (KIA, 2003).
Interaction = A user clicks on an
article known by the personalization system.
6.1 Key figures
The test was performed during
November 19–25, 2001.
There was a total of 81,305
sessions with interactions.
There were 239,676 sessions in
total according to RedSheriff (RedSheriff, 2003), the official
measurement system of web traffic at SvD. Thus, 34% (= 81,305/239,676)
of the sessions at SvD had interactions.
6.2 Result per
personalization group vs. reference
Three different personalization
groups were compared with each other and a non-personalized reference
group with regards to average interactions per session and with a
24-hour sample size (Figure 4).
The 33% session weight
personalization group is between -4.6 % and 11.6% better than the
non-personalized group with regards to average interactions per session.
The average for the week is 1.0% better than the non-personalized group.
The 67% session weight
personalization group is between 1.3 % and 29.5% better than the
non-personalized group with regards to average interactions per session.
The average for the week is 10.0% better than the non-personalized
group.
The 100% session weight
personalization group is between -4.3 % and 19.3% better than the
non-personalized group with regards to average interactions per session.
The average for the week is 8.0% better than the non-personalized group.
Figure 4: The average number of
interactions per session and day for each personalization group.
6.3 Statistical
significance
In order to test the statistical
significance of personalization group differences, a one-way analysis of
variance (ANOVA) was carried out with the number of interactions per
session as the dependent variable and the personalization group as the
group factor (table 2). Prior to carrying out the ANOVA test, a contrast
test was specified between the reference group and all other groups
combined. The ANOVA indicated that the group difference was
statistically significant (F[3,81301]=10.24; p<.001). The contrast
between the reference group and all other groups was also statistically
significant (t= 4.21; DF=51055; p<.001 assuming unequal variances).
Table 2: Means and standard
deviations of interactions per session by groups.
|
Personalization group |
Traffic volume
(sessions) |
Interactions
Mean |
Standard deviation |
|
1_33 |
20,069 |
1.94 |
1.93 |
|
2_67 |
20,339 |
2.12 |
5.21 |
|
3_100 |
20,073 |
2.09 |
6.15 |
|
Reference |
20,824 |
1.93 |
3.18 |
|
Total |
81,305 |
2.00 |
2.17 |
The number of interactions per
session was skewly distributed in all groups, and the assumption of
equality of variances in the ANOVA analysis was violated (Levene's test
of equality of variances[3,81301]=18.16; p<.001). However, a
nonparametric test of group differences, Kruskal-Wallis' test also
showed a significant group difference (c2[3]=30.0; p<.001).
The group differences were further
explored in the number of interactions per session using a
categorization of the number of interactions per session variable into
six categories:
One interaction
Two interactions
Three interactions
4–5 interactions
6–10 interactions
11 or more interactions (385)
Table 3 shows a breakdown of the
number of sessions reduced to six categories, on personalization groups.
The overall effect of personalization on the number of interactions per
session was significant also when categorized in this manner
(c2[15]=59.6; p<.001).
Table 3: Cross-tabulation of number
of sessions in six categories by personalization group.
Number of
interactions |
|
33% |
67% |
100% |
Reference |
Total |
|
1. 1 |
Number of sessions |
11,440 |
11,428 |
11,493 |
12,126 |
46,487 |
|
|
% within group |
57.0% |
56.2% |
57.3% |
58.2% |
57.2% |
|
2. 2 |
Number of sessions |
4,275 |
4,284 |
4,278 |
4,466 |
17,303 |
|
|
% within group |
21.3% |
21.1% |
21.3% |
21.4% |
21.3% |
|
3. 3 |
Number of sessions |
2,034 |
2,165 |
2,031 |
2,055 |
8,285 |
|
|
% within group |
10.1% |
10.6% |
10.1% |
9.9% |
10.2% |
|
4. 4–5 |
Number of sessions |
1,552 |
1,591 |
1,539 |
1,457 |
6,139 |
|
|
% within group |
7.7% |
7.8% |
7.7% |
7.0% |
7.6% |
|
5. 6–10 |
Number of sessions |
687 |
736 |
611 |
616 |
2,650 |
|
|
% within group |
3.4% |
3.6% |
3.0% |
3.0% |
3.3% |
|
6. 11–385 |
Number of sessions |
81 |
135 |
121 |
104 |
441 |
|
|
% within group |
0.4% |
0.7% |
0.6% |
0.5% |
0.5% |
|
Total |
Number of sessions |
20,069 |
20,339 |
20,073 |
20,824 |
81,305 |
|
|
% within group |
100.0% |
100.0% |
100.0% |
100.0% |
100.0% |
If the categorization groups 1 and
2 are combined (1–2 interactions) and compared with categorization
groups 3, 4, 5, and 6 combined (3 or more interactions), then the number
of sessions is distributed as in table 4.
Table 4: Cross-tabulation of number
of sessions in two categories by personalization group.
Number of
interactions |
|
33% |
67% |
100% |
Reference |
Total |
|
1–2 |
Number of sessions |
15,715 |
15,712 |
15,771 |
16,592 |
63,790 |
|
|
% within group |
78.3% |
77.3% |
78.6% |
79.7% |
78.5% |
|
3 or more |
Number of sessions |
4,354 |
4,627 |
4,302 |
4,232 |
17,515 |
|
|
% within group |
21.7% |
22.7% |
21.4% |
20.3% |
21.5% |
|
Total |
Number of sessions |
20,069 |
20,339 |
20,073 |
20,824 |
81,305 |
|
|
% within group |
100.0% |
100.0% |
100.0% |
100.0% |
100.0% |
In the personalized groups it was
between 5.5% and 11.9% more sessions that belonged to the “3 or more
interactions” than the reference group. The personalized groups had a
dislocation towards more interactions per session than the reference
group.
The overall effect of
personalization on the increase of average interactions per session was
thus statistically significant by all of the comparisons made
(parametric, non-parametric, and categorical).
6.4 Turning off the
personalization on the first page
After the test, the personalization
on only the first page was turned off because of internal maintenance at
SvD and the results of the personalized groups compared to the
non-personalized group dropped significantly.
7. Conclusions and discussion
Due to the short test period (one
week) the results should be interpreted carefully and treated only as
indicative.
The test indicates that
personalization works.
It appears as if a medium to high
session-consideration is preferable for obtaining the business objective
of increased traffic volume to the website. The configuration with
greater consideration of the user’s history only performed slighter
better than the non-personalized group. All of the personalized groups
had a dislocation towards more interactions per session than the
reference group.
It appears as if placement on the
web page is an important aspect of successful personalization; if a
personalized function is well exposed it seems like it has a significant
impact on the increase of reading volume.
7.1 Possible problems and
errors
7.1.1 The history of the user
profiles
The user profiles originated from
the beginning of the implementation and were not erased at the start of
the test. My interpretation is that this was positive for the test, as
it would have been difficult to test the impact of history with only
one-week-old user profile data.
7.1.2 Metadata
Possible problems with metadata
coding:
1) Missing metadata. On
occasion, the operators (journalists) did not encode the articles with
metadata. Missing articles are estimated to be less than 0.5% of the
total number of articles.
2) The possibility that the
operators have coded articles wrongly or inconsistently.
3) Inefficient or erroneous
structure of the metadata with regards to the business objectives.
The level of incorrectness in
personalization due to erroneous metadata has not been taken into
account.
7.1.3 Known technical
disturbances
During the test period the
communication between the personalization system and the SvD website was
lost for a few hours on 11/21/2001 due to technical disturbances from
the systems surrounding the SvD website. The impact of these
disturbances is considered minor.
7.1.4 Cookie identification
One problem with anonymous users is
that it is the computer that is identified by a cookie and not the
individual user. This means that the personalization is limited to a
certain computer. Even on the same computer a cookie can be lost to
deletion and thereby limit the effects of the personalization. If a
computer has multiple users, the personalization will be an average of
the behavior of the multiple users. If a single user uses multiple
computers, the personalization system will treat the different computers
as different users.
7.1.5 Search robots
No considerations of the effects on
the results by search robots have been made.
7.2 Future works
Further investigations into the
selection of personalization functions, where they should be placed, how
to optimize the configuration of the personalization system, and finding
an optimal metadata structure are needed both in the news-domain and in
other types of domains. Additional longitudinal tests to confirm the
effects of personalization may be suitable.
8. Acknowledgments
This paper would not have been
possible without the participation of all the employees at Adaptlogic AB
and Johan Möller and Klas Sabelström at Svenska Dagbladet AB.
The statistical tests in the
chapter Statistical significance was carried out with the assistance
from Harald Janson (PhD., Department of Psychology, The University of
Oslo (UiO)).
References
- Chesnais, P. R., Mucklo, M. J.,
Sheena, J. A. (1995) “The Fishwrap Personalized News System” In the
proceedings of the IEEE 2nd International Workshop on Community
Networking Integrating Multimedia Services to the Home, Princetown, NJ.
- Foltz, P.W. and Dumais, S.T.
(1992) “Personalized Information Delivery: An Analysis of Information
Filtering Methods.” Communications of the ACM, Vol 35, No.12, pp51-60.
- Jokela, S., Turpeinen, M., Kurki,
T., Savia, E., Sulonen, R. 2001. The Role of Structured Content in a
Personalized News Service. Proceedings of the 34th Hawaii
International Conference on System Sciences (CD/ROM), January 3–6
2001, Computer Society Press. 12 pages.
- Karben, A. (2003) “Recent
Modifications. Most Recent Changes to the NITF DTD”, [online], News
Industry Text Format, http://www.nitf.org/site/recent-modifications.html
- KIA – Kommitén för
Internetannonsering (2003) Definitioner. http://www.mediacom.it-norr.se/23/kia/final_web/kommentarer.asp
(Accessed 8 December 2003).
- Kristol, D., Montulli, L.,
(1997) “HTTP State Management Mechanism.” Bell Laboratories, Lucent
Technologies, Netscape Communications. RFC 2109.
- Lassila, O. (1997) “Introduction
to RDF Metadata”. W3C – World Wide Web Consortium. http://www.w3c.org/TR/NOTE-rdf-simple-intro-971113.html
(Accessed 8 December 2003).
- Lennstrand, B., Persson, C.,
Wallin, E., (1999) “Design and Prototyping of a Web-based System for
Personalized E-Commerce and Consumer Behavior Research: The Sport
Planet Case.” Presented at COTIM, University of Rhode Island, USA.
- Peppers, D., Rogers, M., and
Dorf, B. (1999) “The OneToOne Fieldbook: the complete toolkit for
implementing a 1to1 marketing program.” ISBN 0-385-49369-X. Doubleday,
New York.
- RedSheriff (2003). http://www.redsheriff.com
(Accessed 8 December 2003).
- Russel, S., Norvig, P., (1995)
“Artificial Intelligence; A Modern Approach.” ISBN 0-13-360124-2.
Prentice Hall, Inc., New Jersey, USA.
- Saunders, C. (2002) “NYTD Points
to ‘Surround Session’ Benefits”. internetnews.com. http://www.internetnews.com/IAR/article.php/12_970991
(Accessed 8 December 2003).
- Stadler, H. Lindgren, J. (2003)
“TT NITF version 3.3“. Tidningarnas Telegrambyrå. http://www.tt.se/tekspec/ttnitfv3_3.doc
(Accessed 8 December 2003).
- SvD (2001). Svenska Dagbladet.
http://www.svd.se (Accessed 8 December 2003).
- Turpeinen, M., Saarela, J.,
Korkea-aho, M., Puskala, T., Sulonen, R. 1996. Architecture for
Agent-Mediated Personalised News Services. PAAM’96: The First
International Conference on Practical Application of Intelligent
Agents and Multi-Agent Technology, 22–24 April, London, UK.
- Wallin, E., Persson, C., (1998)
“Using autonomous agents in Internet marketing; A discussion on
subjective product characteristics in agent assisted trade” in Roger,
J. P., et al (eds.), Technologies for the Information Society;
Developments and Opportunities. IOS Press, Amsterdam, pp225–232.
- Wallin, E., (1999) “Consumer
personalization technologies for e-commerce on the Internet; A
taxonomy.” in Roger, J. P., et al. (eds.), Business and work in the
Information Society; New technologies and applications, IOS Press,
Amsterdam, pp237–242.
- Wallin, E., Persson, C.,
Lennstrand, B., (2000) “Web metrics; Design specifications of
web-based system for personalization with bifurcation” TAGA 2000
Proceedings, Rochester, NY, USA.
-
Winnick Cluts, N. (1997) “An ASP You Can Grasp: The
ABCs of Active Server Pages”. Microsoft Corporation, http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnasp/html/aspover.asp
(Accessed 8 December 2003).
|
