1. Understanding Business¶
Airbnb Inc, is an American online marketplace and hospitality service brokerage company based in San Francisco, California, United States. Members can use the service to arrange or offer lodging, primarily homestays, or tourism experiences.
The booking price of the properties vary according various attributes like its location, property type, size, number of rooms, amenities, day of the week and month of the year etc. Airbnb made its dataset about property listings in Seattle in 2016 publicly available on Kaggle. Our goal is to study this dataset to understand what the major trends in the pricing over the year and across the neighbourhoods in Seattle. We also want to understand through statistical modelling about how each of the attributes related to the property might be influencing the booking price on Airbnb.
2. Understanding Data¶
Importing necessary libraris.
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
import seaborn as sns
from pandas.plotting import register_matplotlib_converters
register_matplotlib_converters()
%matplotlib inline
Reading datasets.
In [2]:
s_calendar = pd.read_csv("./data/seattle/calendar.csv")
s_reviews = pd.read_csv("./data/seattle/reviews.csv")
s_listings = pd.read_csv("./data/seattle/listings.csv")
Understanding the information present in the dataset through the columns present in the datasets.
In [3]:
s_calendar.head()
Out[3]:
In [4]:
s_calendar.shape
Out[4]:
In [5]:
s_listings.head()
Out[5]:
In [6]:
for column in s_listings.columns.tolist():
print(column)
In [7]:
s_listings.shape
Out[7]:
In [8]:
s_reviews.head()
Out[8]:
In [9]:
s_reviews.shape
Out[9]:
3. Preparation of Data¶
In [10]:
s_listings["price"].head()
Out[10]:
In [11]:
s_calendar["price"].head()
Out[11]:
As we can note that the price attribute is supposed to have numerical values but the data type is object because it stores strings with '$' signs. We need to convert this column into numerical data type.
In [12]:
"""
Converts price in the string format like "$1,125.00" into numeric value 1125.00
INPUT:
- string price in string format
OUTPUT:
- float value corresponding to the price or None if the input is not parseable to float
"""
def str_to_num (string):
if string is not None:
if type(string) is str and string.startswith('$'):
return float(string.replace('$', '').replace(',', ''))
else:
return None
else:
return None
In [13]:
s_listings_cleaned = pd.concat([s_listings.drop('price', axis=1), s_listings["price"].apply(str_to_num)], axis=1)
s_listings_cleaned['price'].head()
Out[13]:
In [14]:
s_calendar_cleaned = pd.concat([s_calendar.drop('price', axis=1), s_calendar["price"].apply(str_to_num)], axis=1)
s_calendar_cleaned['price'].head()
Out[14]:
In [15]:
s_calendar_cleaned['available'] = s_calendar_cleaned['available'] == 't'
s_calendar_cleaned['available'].head()
Out[15]:
In [16]:
s_calendar_cleaned["date"].dtype
Out[16]:
In [17]:
s_calendar_cleaned["date"] = pd.to_datetime(s_calendar_cleaned["date"], format="%Y-%m-%d")
In [18]:
s_calendar_cleaned["date"].dtype
Out[18]:
In [19]:
s_calendar_cleaned["date"].head()
Out[19]:
In [20]:
s_listings_cleaned["location_categorical"] = s_listings_cleaned.apply(lambda row: str((format(row.latitude, '.2f'), format(row.longitude, '.2f'))), axis=1)
In [21]:
s_listings_cleaned.location_categorical.head()
Out[21]:
In [22]:
list(map(lambda amenity : amenity.replace("\"","").replace("{","").replace("}", ""), s_listings_cleaned.amenities.iloc[0].split(",")))
Out[22]:
In [23]:
"""
Separates the string value of `amenities` attribute in the row, into a list of individual amenities.
INPUT:
- row : from dataset having amenities attribute
OUTPUT:
- list of amenities derived from the value of `amenities` attribute in row
"""
def separate_amenities(row):
amenities = row.amenities
list_to_return = []
if (amenities is not None and type(amenities) == str):
list_to_return = list(map(lambda amenity : amenity.replace("\"","").replace("{","").replace("}", ""), amenities.split(",")))
if '' in list_to_return:
list_to_return.remove('')
return list_to_return
s_listings_cleaned["amenities_list"] = s_listings_cleaned.apply(separate_amenities, axis=1)
In [24]:
possible_amenities = s_listings_cleaned['amenities_list'].apply(pd.Series).stack().unique()
possible_amenities
Out[24]:
In [25]:
"""
Assigns new boolean attribute to the row based on the presence of that amenity in the list.
Returns updated row with additional attributes corresponding to the amenities added to it.
INPUT:
- row containing attributes related to the property, including `amenities_list`
OUTPUT:
- row containing newly added boolean attributes indicating the presence of each possible type of amenity in the property
"""
def add_amenities_columns (row):
amenities = set(row.amenities_list)
for possible_amenity in possible_amenities:
row["amenity_" + possible_amenity] = possible_amenity in amenities
return row
s_listings_cleaned = s_listings_cleaned.apply(add_amenities_columns, axis=1)
In [26]:
s_listings_cleaned["instant_bookable"] = s_listings_cleaned["instant_bookable"] == "t"
Analytical Questions and Answers¶
1. What is the trend in number of properties becoming available on Airbnb over the year 2016?¶
In [27]:
df = s_calendar_cleaned
df = df.groupby('listing_id', group_keys=False)\
.apply(lambda x: x[x.available.ne(x.available.shift())])\
.reset_index(drop=True)
df.head(20)
Out[27]:
In [28]:
listing_activation_dates = df[df.available == True].groupby("listing_id")[["listing_id","date"]].min()
lower_date = min(s_calendar_cleaned.date)
upper_date = max(s_calendar_cleaned.date)
days = pd.date_range(lower_date, upper_date, freq="D")
number_of_properties_activated_on_date = pd.DataFrame({"date":days, "activated_props": 0}).sort_values(by="date")
number_of_properties_activated_on_date.set_index("date", inplace=True, drop=False)
number_of_properties_activated_on_date_partial = listing_activation_dates.groupby("date")[["listing_id"]].agg(["count"])
number_of_properties_activated_on_date.loc[number_of_properties_activated_on_date_partial.index, "activated_props"] = number_of_properties_activated_on_date_partial.values
In [29]:
number_of_properties_activated_on_date["activations_till_date"] = number_of_properties_activated_on_date["activated_props"].cumsum(axis=0)
# number_of_properties_activated_on_date.head(100)
In [30]:
active_in_2016 = number_of_properties_activated_on_date["activations_till_date"]\
[number_of_properties_activated_on_date["date"] >= pd.to_datetime("2016-01-01")]\
[number_of_properties_activated_on_date["date"] <= pd.to_datetime("2016-12-31")]
dates = active_in_2016.index
months = mdates.MonthLocator()
month_format = mdates.DateFormatter("%b")
fig, ax = plt.subplots()
fig.set_size_inches(12,5)
ax.plot(dates, active_in_2016)
ax.xaxis.set_major_locator(months)
ax.xaxis.set_major_formatter(month_format)
yticks = ax.get_yticks()
ax.set_xlabel("Month in 2016")
ax.set_ylabel("# of properties that became first time available")
ax.set_title("Number of properties that became first time available in 2016")
plt.show()
It is evident from this plot that number of properties that became first time available during 2016 increased rapidly between January to March. After that, the rate of growth slowed down, but the properties becoming first time available during the year continued to grow throughout the year.
2. What is the trend in occupancy of properties over the year 2016?¶
In [31]:
s_calendar_cleaned.groupby("date")[["date", "available"]].agg(["sum"]).head()
Out[31]:
In [32]:
availability_by_date = s_calendar_cleaned.groupby("date")[["date","available"]].agg(["sum"]).sort_values(by="date")
availability_in_2016 = availability_by_date[pd.to_datetime("2016-01-01") <= availability_by_date.index][availability_by_date.index <= pd.to_datetime("2016-12-31")][("available","sum")]
active_in_2016 = number_of_properties_activated_on_date[pd.to_datetime("2016-01-01") <= number_of_properties_activated_on_date.index][number_of_properties_activated_on_date.index <= pd.to_datetime("2016-12-31")]["activations_till_date"]
dates = availability_in_2016.index
occupancy_rate = (1 - availability_in_2016/active_in_2016) * 100
months = mdates.MonthLocator()
month_format = mdates.DateFormatter("%b")
fig, ax = plt.subplots()
fig.set_size_inches(12,5)
ax.plot(dates, occupancy_rate)
ax.xaxis.set_major_locator(months)
ax.xaxis.set_major_formatter(month_format)
yticks = ax.get_yticks()
ax.set_yticklabels(['{}%'.format(y) for y in yticks])
ax.set_xlabel("Month in 2016")
ax.set_ylabel("Occupancy rate")
ax.set_title("Occupancy rate of properties listed on Airbnb in Seattle in 2016")
plt.show()
In [33]:
number_of_occupied_props = (occupancy_rate/100.0) * availability_in_2016
months = mdates.MonthLocator()
month_format = mdates.DateFormatter("%b")
fig, ax = plt.subplots()
fig.set_size_inches(12,5)
ax.plot(dates, number_of_occupied_props)
ax.xaxis.set_major_locator(months)
ax.xaxis.set_major_formatter(month_format)
yticks = ax.get_yticks()
# ax.set_yticklabels(['{}%'.format(y) for y in yticks])
ax.set_xlabel("Month in 2016")
ax.set_ylabel("Number of occupied properties")
ax.set_title("Number of occupied properties listed on Airbnb in Seattle in 2016")
plt.show()
It is evident from the plot that number of poperties occupied remained between 400 to 500. The occupancy jumped significantly in April to reach new level of just above 600. It remained between 600 and 650 during April to July. In July, there was one more surge in the occupancy of properties, which crossed well above 700. After July it started reducing gradually, but still remained well above 650 by the end of the year. Overall, we can claim that increasing number of people occupied properties through Airbnb during 2016 in Seattle.
3. What is the trend in average price of booking properties over the year 2016?¶
In [34]:
avg_price_by_date = s_calendar_cleaned.groupby("date")[["date", "price"]].agg(["mean"])
In [35]:
avg_price_by_date.head(20)
Out[35]:
In [36]:
prices_2016 = avg_price_by_date[avg_price_by_date.index >= pd.to_datetime("2016-01-01")][avg_price_by_date.index <= pd.to_datetime("2016-12-31")]
dates = prices_2016.index
months = mdates.MonthLocator()
month_format = mdates.DateFormatter("%b")
fig, ax = plt.subplots()
fig.set_size_inches(12,5)
ax.plot(dates, prices_2016)
ax.xaxis.set_major_locator(months)
ax.xaxis.set_major_formatter(month_format)
yticks = ax.get_yticks()
# ax.set_yticklabels(['{}%'.format(y) for y in yticks])
ax.set_xlabel("Month in 2016")
ax.set_ylabel("Average booking price in $ of property per night")
ax.set_title("Average booking price of properties listed on Airbnb in Seattle in 2016")
plt.show()
The average price per night of properties listed on Airbnb in Seattle grew steadily between January to July in 2016 from around 125 to 155 USD. It remained high between 150 to 155 USD during summer between July to September. After September, the prices started dropping gradually until it stabilized to somewhere around 140 USD. During the end of December, the prices rised slightly during the Christmas holidays.
In [37]:
prices_in_summer = avg_price_by_date.loc[pd.date_range('2016-07-01', '2016-07-31')]
dates = prices_in_summer.index
days = mdates.DayLocator()
day_format = mdates.DateFormatter("%a")
fig, ax = plt.subplots()
fig.set_size_inches(14,5)
ax.plot(dates, prices_in_summer)
ax.xaxis.set_major_locator(days)
ax.xaxis.set_major_formatter(day_format)
yticks = ax.get_yticks()
ax.set_xlabel("Day in July 2016")
ax.set_ylabel("Average booking price in $ of property per night")
ax.set_title("Average booking price of properties listed on Airbnb in Seattle in July 2016")
plt.show()
Zooming into the average prices in the month of July 2016, we can see that the price spikes on Friday and Saturday nights.
4. What were the types of properties available for booking on Airbnb in Seattle?¶
In [38]:
props_by_type = s_listings_cleaned.dropna(subset=["property_type"]).groupby("property_type")["id"].agg(["count"])
plt.figure(figsize=(14,5))
x = props_by_type.index
y = props_by_type["count"]
plt.bar(x, y)
plt.xticks(x, x, rotation="vertical")
plt.ylabel("Number of properties")
plt.xlabel("Type of property")
plt.title("Distribution of properties based in its type")
plt.show()
As we can see, most of the properties listed in Seattle on Airbnb in 2016 are of type Apartment or House. There are also significant number of properties of type Condominium and Townhouse.
5. Which are the neighbourhoods with largest properties on average available on Airbnb in 2016?¶
In [39]:
s_listings_cleaned.dropna(subset=["square_feet"]).groupby("neighbourhood_cleansed")[["neighbourhood_cleansed", "square_feet"]].agg("mean").sort_values(by="square_feet", ascending=False).rename(index=str, columns={"square_feet": "average square feet"}).head(10)
Out[39]:
North Queen Anne, Fairmount Park, Wallingford and Alki has properties with average size of more than 1400 square feet.
6. Which are the neighbourhoods with highest price on average available on Airbnb in 2016?¶
In [40]:
s_listings_cleaned.dropna(subset=["price"]).groupby("neighbourhood_cleansed")[["neighbourhood_cleansed", "price"]].agg("mean").sort_values(by="price", ascending=False).rename(index=str, columns={"price": "average price per night"}).head(10)
Out[40]:
7. Which are the neighbourhoods with highest number of properties listed on Airbnb in 2016?¶
In [41]:
s_listings_cleaned.groupby("neighbourhood_cleansed")[["neighbourhood_cleansed", "id"]].agg(["count"])["id"].sort_values(by="count", ascending=False).rename(index=str, columns={"count": "listings count"}).head(10)
Out[41]:
8. How does the average booking price vary with the size of the property?¶
In [42]:
df = s_listings_cleaned.dropna(subset=["square_feet", "price"])
x = df.square_feet
y = df.price
plt.scatter(x, y)
plt.xlabel("Property size in square feet")
plt.ylabel("Booking price in USD")
plt.title("Booking price vs size in square feet for property")
plt.show()
It can be seen that for properties with smaller size of upto 800 square feet, prices are highly correlated with size. This means that the prices for similarly sized properties are similar. However, as the property size increases above 800 square feet, the variation of price among similarly sized properties increases significantly.
9. How are different numeric attributes of property correlated to its booking price?¶
In [43]:
df = s_listings_cleaned.dropna(subset=["bedrooms", "price"])
x = df.bedrooms
y = df.price
plt.scatter(x, y)
plt.xlabel("Number of bedrooms in property")
plt.ylabel("Booking price in USD")
plt.title("Booking price against number of bedrooms in property")
plt.show()
In [44]:
df = s_listings_cleaned.dropna(subset=["bathrooms", "price"])
x = df.bathrooms
y = df.price
plt.scatter(x, y)
plt.xlabel("Number of bathrooms in property")
plt.ylabel("Booking price in USD")
plt.title("Booking price against number of bathrooms in property")
plt.show()
9. Can we predict the price of booking a property on given night based on its attributes?¶
In this case price is the target value. We can drop the rows with missing values in this target variable for the purpose of training and evaluating our model.
In [45]:
input_vars_from_listings = ["neighbourhood_cleansed",
"neighbourhood_group_cleansed",
"city",
"state",
"zipcode",
"market",
"location_categorical",
"property_type",
"room_type",
"accommodates",
"bathrooms",
"bedrooms",
"beds",
"bed_type",
"square_feet",
"guests_included",
"minimum_nights",
"maximum_nights",
"review_scores_rating",
"review_scores_accuracy",
"review_scores_cleanliness",
"review_scores_checkin",
"review_scores_communication",
"review_scores_location",
"review_scores_value",
"jurisdiction_names",
"cancellation_policy",
"reviews_per_month",
"number_of_reviews"
]
amenity_variables = list(map(lambda amenity : "amenity_" + amenity, possible_amenities))
input_vars_from_listings.extend(amenity_variables)
In [46]:
s_calendar_cleaned["month"] = s_calendar_cleaned['date'].dt.month_name()
In [47]:
s_calendar_cleaned["day_of_week"] = s_calendar_cleaned['date'].dt.weekday_name
In [48]:
df = pd.merge(s_listings_cleaned, s_calendar_cleaned, left_on="id", right_on="listing_id", how="inner", suffixes=("_listings", "_calendar"))
In [49]:
df = df.dropna(subset=["price_calendar"])
In [50]:
sns.heatmap(df[["price_calendar",
"square_feet",
"bedrooms",
"bathrooms",
"accommodates",
"beds",
"guests_included"
]].rename(columns={"price_calendar": "price"}).corr(), annot=True, fmt=".2f");
In [51]:
input_vars_from_calendar = ["month", "day_of_week"]
In [52]:
all_input_vars = input_vars_from_listings + input_vars_from_calendar;
In [53]:
X = df[all_input_vars]
y = df["price_calendar"]
The categorical variables are converted into dummy numerical variables by adding one column per categorical level of each categorical attribute using pandas get_dummies function.
We are not replacing the missing values in categorical attributes as it will be handled during the creation of dummy numerical variables by creation of separate numerical column representing the NaN value of that attribute.
In [54]:
cat_vars = X.select_dtypes(include=['object']).copy().columns
for var in cat_vars:
# for each cat add dummy var, drop original column
X = pd.concat([X.drop(var, axis=1), pd.get_dummies(X[var], prefix=var, prefix_sep='_', drop_first=False, dummy_na=True)], axis=1)
We need to prepare the data by replacing missing values with appropriate values in input variables. In the case of numerical attributes, we are replacing them with the mean of the column.
In [55]:
def fill_mean (col):
return col.fillna(col.mean())
X = X.apply(fill_mean, axis=0)
In [57]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .30, random_state=42)
In [58]:
lm_model = Ridge(normalize=True,fit_intercept=True)
In [59]:
lm_model.fit(X_train, y_train)
Out[59]:
In [60]:
y_test_preds = lm_model.predict(X_test)
In [61]:
"The r-squared score for the model {} on {} values.".format(r2_score(y_test, y_test_preds), len(y_test))
Out[61]:
In [62]:
plt.figure(figsize=(12,5))
plt.scatter(y_test, y_test_preds)
plt.xlabel("Y test")
plt.ylabel("Y test preds")
plt.show()
In [63]:
def coef_weights(coefficients, X_train):
'''
INPUT:
coefficients - the coefficients of the linear model
X_train - the training data, so the column names can be used
OUTPUT:
coefs_df - a dataframe holding the coefficient, estimate, and abs(estimate)
Provides a dataframe that can be used to understand the most influential coefficients
in a linear model by providing the coefficient estimates along with the name of the
variable attached to the coefficient.
'''
coefs_df = pd.DataFrame()
coefs_df['input_variable'] = X_train.columns
coefs_df['coef'] = lm_model.coef_
coefs_df['abs_coef'] = np.abs(lm_model.coef_)
coefs_df = coefs_df.sort_values('abs_coef', ascending=False)
return coefs_df
In [64]:
coef_df = coef_weights(lm_model.coef_, X_train)
In [65]:
coef_df.head(50)
Out[65]:
Influence of neighbourhood on price¶
In [66]:
coef_df[coef_df["input_variable"].str.startswith("neighbourhood")]
Out[66]:
Observing the coefficients of input variables other than those related to the location of property¶
In [67]:
coef_df[~coef_df["input_variable"].str.startswith("location") & ~coef_df["input_variable"].str.startswith("neighbourhood") & ~coef_df["input_variable"].str.startswith("zipcode")]
Out[67]:
Influence of Month on price¶
In [68]:
coef_df[coef_df["input_variable"].str.startswith("month")]
Out[68]:
Influence of Day of week on price¶
In [69]:
coef_df[coef_df["input_variable"].str.startswith("day_of_week")]
Out[69]:
Influence of Property type on price¶
In [70]:
coef_df[coef_df["input_variable"].str.startswith("property_type")]
Out[70]:
Influence of Amenities on price¶
In [71]:
coef_df[coef_df["input_variable"].str.startswith("amenity")]
Out[71]:
Observing the coefficients, there are several conclusions that can be made:¶
- The properties of type Boat are very likely to have higher booking price
- The dorm, shared room or tent type properties are very likely to have lower booking price
- The precise location and neighbourhood that the property is located play very important role in deciding its booking price.
- The number of bathrooms and bedrooms in the property are in strong positive proportion with the booking price.
- The prices are also in positive proportion with the number of people it accommodates.
- The amenities that strongly and positively influence the booking price are having doorman, pool, cable TV, hot tub, gym and elevator in the building.
- The booking price tends to be higher in months July and August, while it tends to be lower in January and February.
- Booking prices for Friday and Saturday tends to be higher relative to other days.
- Permission of smoking as amenity has strong influence on the price on the negative side.
- Properties located in Phinney Ridge Seattle city are less likely to have high price as per this trained model on the available data.
- As all the properties in the dataset are located in Seattle, the attributes like market and jurisdiction are not playing any role in determining the price as they are not adding any variance in the input data. However, they might be of significance while training this same model on dataset containing listings across multiple cities and jurisdictions in the U.S.